Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors

# Explicit Fuzzy Logic in the Feed-Forward Layer Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
Source: [https://arxiv.org/html/2606.31845](https://arxiv.org/html/2606.31845)
Mark Oskin Professor School of Computer Science and Engineering University of Washington mhoskin@uw\.edu

\(June 2026\)

###### Abstract

The feed\-forward \(FFN\) sublayer is where a transformer materializes the conjunctions and distinctions its attention has gathered—yet its standard form, a pointwise activation of a linear projection, exposes no account of*what*it computes\. We study a parameter\-neutral replacement in which each hidden unit is an explicit fuzzy set operation on sigmoid\-bounded\[0,1\]\[0,1\]membership values: intersectionA⋅BA\\cdot Band set\-differenceA⋅\(1−B\)A\\cdot\(1\-B\), the latter a bounded,*positive*encoding of negation \(“AAbut notBB”\) that a gated or bilinear unit structurally lacks\. We call this a negation\-capable FFN \(NC\-FFN\)\. On a controlled reasoning probe \(NN\-bit parity in pure feed\-forward stacks\), bounded multiplicative units are the most parameter\-efficient reasoning basis at shallow depth; at language\-model scale \(125M parameters, OpenWebText\),NC\-FFNties the GELU baseline’s perplexity at equal parameters, with every unit carrying explicit logical form\. But two limits emerge from the same cause\. Genuine two\-operand logic localizes sharply to the embedding\-adjacent layer and*erodes*toward soft gating under language\-model training; and the model’s one robust grammatical deficit is concentrated in*licensing*and*quantifiers*—precisely the sequence\-level constructions that within\-token set operators cannot express\. Our central result resolves both at once\. We add a small, parameter\-neutral block of fuzzy*quantifiers*over the sequence—a soft existential and a soft proportion, each with a per\-unit*learned forgetting rate*initialized at the non\-forgetting \(sticky\) limit\. This recovers the grammatical deficit at epoch one—halving the wider gap that opens by epoch two—and modestly leads the baseline on LAMBADA, while making the FFN markedly more legible\. The logical structure now*holds and migrates into depth*rather than eroding; the learned decay*un\-learns its own stickiness*—every unit forgets within a few tokens \(median half\-life∼\\sim1\.51\.5;*zero*near\-permanent units\)—so the quantifier is a local, predictive operator, not a latch; and at the semantic layers the units read, without dictionary learning, as grammatical*licensing detectors*\. Each fires on a grammatical*licensor*\(a comparative, a passive participle, a negative\-polarity item\) and its∼\\sim1\.51\.5\-token memory carries the membership forward to predict the licensed function word \(*than*,*by*,*nor*\)—the decay having tuned itself to the grammatical\-licensing window\. This legibility is localized \(a few percent of the network, concentrated at the late layers\) and free only up to a partition: a*fully*Boolean FFN with no stabilizing GELU fraction still diverges in training, leaving an end\-to\-end\-legible model an open optimization problem\. But within those bounds the result is a parameter\-neutral, language\-model\-quality transformer that contains a readable, interpretable\-by\-construction grammatical mechanism—an explicit account not just of*what*a feed\-forward layer represents but of*how*it licenses\.

## 1Introduction

Mechanistic interpretability has made the most progress where computation is structurally exposed: attention patterns are legible because attention is, by construction, a weighted routing of values, and induction or copy circuits can be read off the QK/OV structure\. The feed\-forward sublayer, which holds the majority of a transformer’s parameters and is widely argued to be where factual and feature\-level computation lives\(Gevaet al\.,[2021](https://arxiv.org/html/2606.31845#bib.bib23)\), is far less transparent: a pointwise activation of a linear projection is a dense, polysemantic, sign\-agnostic object whose*operation*on its inputs has no native reading\. Post\-hoc dictionary methods recover*what*directions a layer represents\(Brickenet al\.,[2023](https://arxiv.org/html/2606.31845#bib.bib21); Cunninghamet al\.,[2024](https://arxiv.org/html/2606.31845#bib.bib22)\), but not*how*the layer combines them\.

We take a complementary, architectural route: choose an FFN whose*combination rule is explicit*, and ask what that costs and reveals\. The FFN is a natural home for set logic—it is where attention’s gathered features are conjoined into “this token is a verb*and*the subject is plural” or “a quotation*but not*a heading\.” The last example is the crux\. Negation is cheap to*state*but awkward to*represent*: a raw feature direction has no notion of “absence,” so “notBB” has no clean positive form\. The fix, known from logical\-query embeddings over knowledge graphs\(Ren and Leskovec,[2020](https://arxiv.org/html/2606.31845#bib.bib14)\), is to make features*bounded*memberships in\[0,1\]\[0,1\], a space closed under complement, so that “notBB” is simply1−B1\-B\. Embedding the set operators of fuzzy logic— intersectionA⋅BA\\cdot B, set\-differenceA⋅\(1−B\)A\\cdot\(1\-B\), with the complement supplying negation—into the transformer FFN gives a layer in which every hidden unit is a namedand/and\-notof two operands, at no change to the parameter budget, head configuration, or attention\. We call it a negation\-capable FFN \(NC\-FFN\); “capable” is an architectural property, not a performance promise\.

The operators, the multiplicative\-FFN form \(GLU and bilinear variants\), the operator\-softening optimization pathology, and the negation\-via\-complement idea are each established prior art \(Section[2](https://arxiv.org/html/2606.31845#S2)\); our contribution is their assembly inside a transformer language model and an empirical account of what it does\. The account has five parts: four that characterizeNC\-FFNitself, and a fifth that extends it with a sequence\-level operator and yields the paper’s headline result\.

#### Capability \(Section[5](https://arxiv.org/html/2606.31845#S5)\)\.

The motivating question—does a more*structured*component reason more*compactly*?—is not answerable from perplexity, which rewards distributional fit, not reasoning\. We isolate it with a controlled probe:NN\-bit parity, the canonical function that is easy for products and hard for sums, learned by pure feed\-forward stacks of each FFN type\. Measuring the largestNNeach solves as a function of width and depth turns “reasoning capacity” into a number\. The result is a clean dissociation:*bounded*multiplicative units \(NC\-FFN, sigmoid\-bilinear\) are the most parameter\-efficient reasoning basis at shallow depth—a single layer outreaches strictly larger GELU—while*unbounded*bilinear products are useless shallow but compose through depth\. The two properties our architecture toggles, multiplicativity and bounding, turn out to control two different axes of reasoning efficiency\. The small networks here are a controlled instrument for measuring capability, not a scale claim\.

#### Language modeling \(Section[6](https://arxiv.org/html/2606.31845#S6)\)\.

As a 125M\-parameter language model,NC\-FFNis a faithful parameter\-neutral drop\-in: it ties the GELU baseline on perplexity \(and the tie tightens with training\), with a small, persistent deficit on grammatical structure \(BLiMP\)\. We attribute the deficit not to a missing capability but to capacity*allocation*: a bounded multiplicative unit exposes fewer independent standalone features per parameter than an unbounded activation, and ordinary language wants standalone features\. The capability the architecture buys is real but*dormant*for next\-token prediction, which does not call for it\.

#### Legibility and its dynamics \(Section[7](https://arxiv.org/html/2606.31845#S7)\)\.

EveryNC\-FFNunit carries explicit logical form, and a small subset reads as recognizable predicates with no dictionary learning\. Genuine two\-operand logic localizes sharply to the embedding\-adjacent layer; the features the readable units track are*causally used*even though the units are individually redundant\. Most strikingly, the network’s logical content is*dynamic and task\-shaped*: trained on a task that rewards multiplicative reasoning, the Boolean structure*crystallizes*precisely at the grokking transition; trained as a language model, which does not reward it, the same structure*erodes*toward soft gating\. The architecture keeps the logic the task pays for\.

#### A trainability boundary \(Section[8](https://arxiv.org/html/2606.31845#S8)\)\.

The stabilizingGELU\\operatorname\{GELU\}majority beside the Boolean block is not optional\. As the Boolean fraction rises the model trains longer but then diverges sooner, and a*fully*Boolean FFN diverges within the first∼\\sim1616k steps; the obvious remedies—bounding the residual write, a parallel linear highway—only delay it, implicating a saturated\-product gradient pathology rather than the write magnitude\. Legibility is thus free only up to a partition, and an end\-to\-end\-legible model is gated on removing this instability\.

#### Self\-forgetting quantifiers and legible licensing \(Section[9](https://arxiv.org/html/2606.31845#S9)\)\.

Two of the findings above are limitations with a shared cause: the two\-operand logic erodes and localizes to layer 0, and the grammatical deficit concentrates in*licensing*and*quantifiers*— constructions about whether a feature occurred*earlier in the sequence*, whichNC\-FFN’s within\-token operators cannot express\. We add the missing primitive: a parameter\-neutral block of fuzzy quantifiers over the sequence \(a soft existential and a soft proportion\) with a per\-unit*learned forgetting*rate, initialized at the non\-forgetting \(sticky\) limit\. It resolves both limitations at once\. It recovers the grammatical deficit at epoch one—halving the wider epoch\-two gap—and leads the baseline on LAMBADA; the logical structure now*holds and migrates into depth*instead of eroding; the learned decay*un\-learns*its sticky initialization—every unit forgets within a few tokens—so the quantifier is a local, predictive operator rather than a latch; and at the semantic layers the units read, without dictionary learning, as grammatical*licensing detectors*, each firing on a grammatical licensor and carrying it forward—on a∼\\sim1\.51\.5\-token memory that the decay has tuned to the licensing window—to predict the licensed function word\. The legibility is localized, not network\-wide, but it is a readable, interpretable\-by\-construction grammatical mechanism at no parameter or language\-model\-quality cost\.

Together these say that an explicit\-combination FFN is a viable, parameter\-neutral component that reasons more compactly on tasks that need it and exposes*how*it computes—at a modest, well\-localized cost; that “how much logic” is not a fixed property of the architecture but a readout of what its training objective rewards; and that giving the feed\-forward layer an explicit*sequence*operator, free to learn how long to remember, turnsNC\-FFN’s hardest grammatical failures into its most legible mechanism—a bank of self\-forgetting quantifiers that compute readable grammatical licensing\.

## 2Related Work

NC\-FFNsits at the intersection of several mature lines of work: gated and multiplicative feed\-forward layers, differentiable fuzzy logic, set\-operator embeddings for logical query answering, and the interpretability of transformer feed\-forward \(FFN\) sublayers\. The individual ingredients are all established\. Our contribution is one of*placement and lens*: assembling sigmoid\-bounded fuzzy set operators—including an explicit complement/negation term—into a parameter\-neutral FFN sublayer of a decoder\-only language model, and characterizing what that choice does\. We organize the prior art accordingly and state, for each line, the precise gap that remains\.

### 2\.1Gated and multiplicative feed\-forward layers

Gated Linear Units \(GLUs\) replace a feed\-forward activation with the component\-wise product of two linear projections, one passed through a gate\(Dauphinet al\.,[2017](https://arxiv.org/html/2606.31845#bib.bib3)\)\.Shazeer \([2020](https://arxiv.org/html/2606.31845#bib.bib4)\)systematized GLU variants for the transformer FFN—GEGLU, SwiGLU, and a purely*bilinear*variant\(x​W\)⊙\(x​V\)\(xW\)\\odot\(xV\)with no element\-wise nonlinearity—and showed they improve language\-model quality at matched parameter count\. These are the closest architectural relatives ofNC\-FFN: a GLU is itself a sigmoid\-gated multiplicative unit\. The distinction is structural\. A GLU fuses a gate and a value into a*single*multiplicatively\-gated pathway, whereasNC\-FFNplaces a standard activation block*beside*a separate block of explicit set operations and reads them out through a shared projection; and crucially, no GLU variant carries a complement/negation \(1−B1\-B\) term or a set\-theoretic reading of its products\.

Pearceet al\.\([2025](https://arxiv.org/html/2606.31845#bib.bib5)\)andPearceet al\.\([2024](https://arxiv.org/html/2606.31845#bib.bib6)\)study the bilinear FFNg​\(x\)=\(W1​x\)⊙\(W2​x\)g\(x\)=\(W\_\{1\}x\)\\odot\(W\_\{2\}x\)in transformer language models specifically as a substrate for*weight\-based*mechanistic interpretability: because the layer is a pure bilinear form with no nonlinearity, each output is a quadratic form whose interaction structure can be recovered by eigendecomposition\. They report that bilinear MLPs are a competitive, parameter\-neutral drop\-in for activation MLPs\. This is the single nearest piece of prior art to ours, and we treat it as both the behavioral control and the head\-to\-head interpretability comparison \(Sections[4](https://arxiv.org/html/2606.31845#S4),[7](https://arxiv.org/html/2606.31845#S7)\)\. The differences are specific and load\-bearing: bilinear operands are unbounded and signed rather than sigmoid\-bounded to\[0,1\]\[0,1\]; there is no complement/negation term; and there is no set\-operator semantics\. As we show, these are not cosmetic: the sigmoid bounding givesNC\-FFNa graceful single\-operand fallback that a raw bilinear unit structurally lacks\.

Neural Arithmetic Logic Units\(Trasket al\.,[2018](https://arxiv.org/html/2606.31845#bib.bib8)\)use a sigmoid gate to select between additive and multiplicative pathways for systematic numerical extrapolation—adjacent gated\-multiplicative machinery, but aimed at arithmetic rather than set logic and not used as a general FFN\.

### 2\.2Differentiable fuzzy logic and logic\-gate networks

The operatorsNC\-FFNuses are textbook continuous \(fuzzy\) relaxations of Boolean connectives\. The product t\-norm realizes conjunction asA⋅BA\\cdot B, the strong complement realizes negation as1−A1\-A, and set\-difference follows asA⋅\(1−B\)A\\cdot\(1\-B\)\. Neuro\-symbolic frameworks such as Logic Tensor Networks\(Badreddineet al\.,[2022](https://arxiv.org/html/2606.31845#bib.bib10)\)and Logical Neural Networks\(Riegelet al\.,[2020](https://arxiv.org/html/2606.31845#bib.bib11)\)build differentiable logical formulae from such operators, andvan Kriekenet al\.\([2022](https://arxiv.org/html/2606.31845#bib.bib9)\)analyze their optimization behavior, showing that gradient descent tends to*soften*crisp fuzzy operators because product\-form gates suffer vanishing gradients as operands saturate\. Differentiable logic\-gate networks\(Petersenet al\.,[2022](https://arxiv.org/html/2606.31845#bib.bib12),[2024](https://arxiv.org/html/2606.31845#bib.bib13)\)learn, per neuron, one of the sixteen two\-input Boolean functions via temperature\-annealed relaxations, and achieve strong efficiency on vision benchmarks\. These works establish the operators and their optimization pathologies, but always as a*standalone*logical network \(typically for vision or tabular data\), never as an FFN sublayer inside a transformer trained for language modeling\.van Kriekenet al\.\([2022](https://arxiv.org/html/2606.31845#bib.bib9)\)in particular predicts—in non\-transformer settings—the softening we observe; our depth\-resolved measurements \(Section[7](https://arxiv.org/html/2606.31845#S7)\) make this concrete inside a language model and localize where the softening does*not*happen\. Pushed past a roughly even split toward a fully Boolean layer, this same saturation pathology no longer merely*softens*the operators but destabilizes training outright \(Section[8](https://arxiv.org/html/2606.31845#S8)\)\.

### 2\.3Quantifiers, aggregation, and learned forgetting

The within\-token set operators above have a natural sequence extension—*quantifiers*—and our self\-forgetting quantifier block \(Section[9](https://arxiv.org/html/2606.31845#S9)\) draws on three established lines\. First,*quantifier aggregation*in differentiable logic: fuzzy and neuro\-symbolic frameworks realize the existential as a maximum \(or a softpp\-mean/log\-sum\-exp\) and the universal as a mean or product over a domain\. Logic Tensor Networks\(Badreddineet al\.,[2022](https://arxiv.org/html/2606.31845#bib.bib10)\)and Logical Neural Networks\(Riegelet al\.,[2020](https://arxiv.org/html/2606.31845#bib.bib11)\)build exactly these quantifier aggregations, andvan Kriekenet al\.\([2022](https://arxiv.org/html/2606.31845#bib.bib9)\)analyze their gradients\. Our existential \(a leaky cumulative maximum\) and proportion \(a running mean\) are the causal, left\-to\-right specializations of those aggregations over the prefix of the sequence\. Second,*learned forgetting*: a decayed cumulative max or moving average with a*learned*per\-unit rate is, mechanistically, the forget gate of gated recurrent networks\(Gerset al\.,[2000](https://arxiv.org/html/2606.31845#bib.bib31)\)—a leaky integrator whose time constant is trained rather than fixed\. Our contribution is to give a fuzzy*quantifier*this learned temporal scope, so the operator can learn its own window and, as we find,*un\-learn*a sticky cumulative initialization toward a short, predictive memory; to our knowledge prior fuzzy quantifiers aggregate over a fixed or unbounded domain without a learned decay, and none has been placed as an FFN sublayer of a language model\. Third,*grammatical licensing*: the constructions our units detect—NPI licensing, comparatives, passives—are the long\-distance dependencies that targeted grammatical evaluations such as BLiMP\(Warstadtet al\.,[2020](https://arxiv.org/html/2606.31845#bib.bib25)\)probe, and that are inherently about whether a licensor occurred earlier in the string\. Where the interpretability of licensing is otherwise a post\-hoc reading of an opaque model, our quantifier units realize a licensing detector*by construction*\.

### 2\.4Set operators and negation in embedding space

A parallel line uses fuzzy/box/cone set operators to answer logical queries over knowledge graphs\. BetaE\(Ren and Leskovec,[2020](https://arxiv.org/html/2606.31845#bib.bib14)\)represents entities as Beta distributions and implements conjunction, disjunction, and—notably—*negation*as a closed operation, motivated by queries of the form “XXandYYbut notZZ\.” Query2box\(Renet al\.,[2020](https://arxiv.org/html/2606.31845#bib.bib15)\), ConE\(Zhanget al\.,[2021](https://arxiv.org/html/2606.31845#bib.bib16)\), and FuzzQE\(Chenet al\.,[2022](https://arxiv.org/html/2606.31845#bib.bib17)\)develop related set\-operator embeddings, the last using explicit fuzzy logic\. These works are the origin of the complement\-for\-negation idea we adopt, and their motivating use case is essentially identical to ours at the conceptual level\. The gap is again placement: the operators live in a dedicated*query\-answering head*over a knowledge graph, not as a general\-purpose FFN in a language model trained on next\-token prediction\. An adversarial citation\-forward sweep from this literature did not surface any migration of the operators into an LM FFN\.

### 2\.5Interpretability of feed\-forward sublayers

A substantial body of work treats the FFN as the locus of interpretable computation:Gevaet al\.\([2021](https://arxiv.org/html/2606.31845#bib.bib23)\)cast FFN layers as key\-value memories, Softmax Linear Units\(Elhageet al\.,[2022a](https://arxiv.org/html/2606.31845#bib.bib18)\)modify the activation to encourage more interpretable, less polysemantic units, and Codebook Features\(Tamkinet al\.,[2023](https://arxiv.org/html/2606.31845#bib.bib19)\)bind hidden states to a small set of discrete, nameable codes via a vector\-quantization bottleneck—the*interpretable\-by\-construction*intervention closest in spirit to ours, though it constrains the activation’s discreteness rather than, asNC\-FFNdoes, the layer’s combination rule\. More broadly, individual neurons are known to be*polysemantic*due to superposition\(Elhageet al\.,[2022b](https://arxiv.org/html/2606.31845#bib.bib20)\), which has motivated dictionary\-learning methods—sparse autoencoders— that recover more monosemantic features post hoc\(Brickenet al\.,[2023](https://arxiv.org/html/2606.31845#bib.bib21); Cunninghamet al\.,[2024](https://arxiv.org/html/2606.31845#bib.bib22)\)\.NC\-FFNrelates to this line in two ways\. First, likePearceet al\.\([2025](https://arxiv.org/html/2606.31845#bib.bib5)\), it is an architecture chosen partly for interpretability; but where bilinear interpretability is a global, weight\-based spectral decomposition,NC\-FFNexposes*explicit logical form*—each unit is anand/and\-notof two operands, including a negation a bilinear eigenvector cannot express\. Second, our findings are consistent with superposition: the*operands*ofNC\-FFNunits are themselves distributed/polysemantic directions, so the architecture supplies the combination rule for free but not, in general, monosemantic predicates—a small directly\-readable subset notwithstanding \(Section[7](https://arxiv.org/html/2606.31845#S7)\)\.

### 2\.6Neuro\-fuzzy and fuzzy\-membership transformers

Finally, an applied literature combines fuzzy logic with transformers, largely in forecasting, fault diagnosis, and control\. The nearest work to ours on keywords isHuang and Raza \([2025](https://arxiv.org/html/2606.31845#bib.bib24)\), which augments a transformer language model with a parallel fuzzy\-*membership*feature channel \(interpretable per\-token features such as part\-of\-speech cues, sentiment polarity, and boundary flags\) graded by differentiable membership functions and fused via a gated adapter with an auxiliary reconstruction loss\. This shares the words “fuzzy,” “membership,” and “language model,” but the mechanism is a*side\-channel adapter*that injects hand\-specified features, not a set\-operator FFN computed from the residual stream\. It does not implement set operators, has no complement/negation term, and does not replace the FFN\.

### 2\.7Summary

Taken together, the operators \(fuzzy t\-norms with complement\), the multiplicative\-FFN form \(GLU, bilinear\), the optimization pathology \(operator softening\), and the negation\-via\-complement motivation \(KG query embeddings\) are all prior art that we build on\. What is, to our knowledge, novel is the specific assembly:*sigmoid\-bounded fuzzy set operators, including an explicit complement/negation term, as a parameter\-neutral feed\-forward sublayer of a decoder\-only language model*—and, as the paper’s headline extension, a*self\-forgetting fuzzy quantifier*over the sequence \(a causal existential/proportion with a per\-unit learned forgetting rate\) as a sublayer of the same model, which we find computes legible grammatical licensing\. The defensible fingerprint that distinguishesNC\-FFNfrom its nearest neighbors is the explicit\(1−B\)\(1\-B\)complement inside a transformer FFN: bilinear MLPs have the multiplicative structure but no bounding and no complement\(Shazeer,[2020](https://arxiv.org/html/2606.31845#bib.bib4); Pearceet al\.,[2025](https://arxiv.org/html/2606.31845#bib.bib5)\); GLUs have the sigmoid gate but no set\-operator framing\(Shazeer,[2020](https://arxiv.org/html/2606.31845#bib.bib4)\); differentiable logic networks have the operators but are not transformer LMs\(Petersenet al\.,[2024](https://arxiv.org/html/2606.31845#bib.bib13)\); the KG\-query embeddings have the operators and negation but in a query head\(Ren and Leskovec,[2020](https://arxiv.org/html/2606.31845#bib.bib14)\); and the nearest fuzzy\-membership LM\(Huang and Raza,[2025](https://arxiv.org/html/2606.31845#bib.bib24)\)is a side\-channel adapter rather than an FFN operator\. Our aim is therefore not a new operator but a new view of an old one inside the transformer FFN, and an empirical account of what that view reveals\.

## 3TheNC\-FFNLayer

### 3\.1Background: the transformer FFN

A standard transformer feed\-forward sublayer mapsx∈ℝdx\\in\\mathbb\{R\}^\{d\}through an up\-projection, an element\-wise activation, and a down\-projection:

FFN​\(x\)=Wo​GELU⁡\(Win​x\),Win∈ℝd𝑓𝑓×d,Wo∈ℝd×d𝑓𝑓,\\mathrm\{FFN\}\(x\)=W\_\{o\}\\,\\operatorname\{GELU\}\(W\_\{\\mathrm\{in\}\}\\,x\),\\qquad W\_\{\\mathrm\{in\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathit\{ff\}\}\\times d\},\\;\\;W\_\{o\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathit\{ff\}\}\},\(1\)with2​d​d𝑓𝑓2\\,d\\,d\_\{\\mathit\{ff\}\}parameters\.NC\-FFNreplaces the activation with explicit fuzzy set operations while preserving this parameter budget exactly\.

### 3\.2Fuzzy set operators as hidden units

We read each hidden unit as a fuzzy set\-membership value in\[0,1\]\[0,1\]\. From the layer input we form two membership vectors via independent linear projections and a logistic nonlinearity,

A=σ​\(Wa​x\),B=σ​\(Wb​x\),Wa,Wb∈ℝm×d,A=\\sigma\(W\_\{a\}\\,x\),\\qquad B=\\sigma\(W\_\{b\}\\,x\),\\qquad W\_\{a\},W\_\{b\}\\in\\mathbb\{R\}^\{m\\times d\},\(2\)and combine them with product\-form fuzzy operators\. The default configuration uses intersection and set\-difference,

h∩=A⊙B\(“​A​and​B​”\),h∖=A⊙\(1−B\)\(“​A​and not​B​”\),h\_\{\\cap\}=A\\odot B\\quad\(\\text\{\`\`\}A\\text\{ and \}B\\text\{''\}\),\\qquad h\_\{\\setminus\}=A\\odot\(1\-B\)\\quad\(\\text\{\`\`\}A\\text\{ and not \}B\\text\{''\}\),\(3\)where⊙\\odotis the element\-wise product\. The complement1−B1\-Bsupplies an explicit, bounded,*positive*encoding of negation: a unit can fire for the presence of one feature and the*absence*of another\. This explicit negation primitive is the architectural feature that distinguishes the layer from its multiplicative neighbors—a bilinear or gated unit has no clean*positive*encoding of “AAand notBB”—and we accordingly call it a*negation\-capable*FFN \(NC\-FFN\)\. We stress that “capable” denotes an*architectural*property, not a performance claim\. A standard FFN can*approximate*negated functions; whatNC\-FFNadds is an explicit, bounded, interpretable negation primitive\. And as Section[7](https://arxiv.org/html/2606.31845#S7)shows, possessing the primitive does not by itself improve—indeed it slightly degrades—performance on negation\-sensitive grammar: the name denotes the capability, not its benefit\. The combined hidden vector is read out through a single down\-projection,

NC\-FFN​\(x\)=Wo​h,h=\[h∩;h∖\]∈ℝ2​m,Wo∈ℝd×2​m\.\\mbox\{NC\-FFN\}\(x\)=W\_\{o\}\\,h,\\qquad h=\[\\,h\_\{\\cap\}\\,;\\,h\_\{\\setminus\}\\,\]\\in\\mathbb\{R\}^\{2m\},\\qquad W\_\{o\}\\in\\mathbb\{R\}^\{d\\times 2m\}\.\(4\)We refer to a paired\(Ai,Bi\)\(A\_\{i\},B\_\{i\}\)and the operations built from it as a*unit*, and to the explicitand/and\-notstructure as the unit’s logical*form*\.

### 3\.3Parameter neutrality

Forkkoperators built from a single\(A,B\)\(A,B\)pair, the input side comprisesWa,WbW\_\{a\},W\_\{b\}\(2​m2mrows\) and the output sideWoW\_\{o\}hasH=k​mH=kmcolumns, giving\(2​m\+k​m\)​d=\(2\+k\)​m​d\(2m\+km\)\\,d=\(2\+k\)\\,m\\,dparameters\. A pureNC\-FFNlayer withkkoperators is therefore parameter\-matched to aGELU\\operatorname\{GELU\}FFN when

\(2\+k\)​m=2​d𝑓𝑓\.\(2\+k\)\\,m=2\\,d\_\{\\mathit\{ff\}\}\.\(5\)ThusNC\-FFNdoes not change the layer’s parameter count, head configuration, or attention; it spends the same budget on constrained multiplicative set operations rather than free, unbounded activations\. Because the bounded operators occupy capacity aGELU\\operatorname\{GELU\}block would otherwise have, the*effective*flexibility per unit is lower; whether this trades favorably is an empirical question \(Section[7](https://arxiv.org/html/2606.31845#S7)\)\.

### 3\.4The hybrid partition

A pureNC\-FFNlayer is difficult to train\. As operands saturate, the product\-form gradients vanish \(the pathology analyzed byvan Kriekenet al\.\([2022](https://arxiv.org/html/2606.31845#bib.bib9)\)\), and an all\-positive, bounded basis is a poor substrate for the output projection; in our experiments a pure layer trains normally for thousands of steps and then diverges abruptly, a conditional instability we characterize in Section[8](https://arxiv.org/html/2606.31845#S8)\. We therefore use a*hybrid partition*: a fraction of the layer budget is given to a standardGELU\\operatorname\{GELU\}“gradient highway” and the remainder to the set\-operator block, read out through one sharedWoW\_\{o\}:

h=\[GELU⁡\(Wg​x\);h∩;h∖\],Wg∈ℝg×d\.h=\[\\,\\operatorname\{GELU\}\(W\_\{g\}\\,x\)\\,;\\,h\_\{\\cap\}\\,;\\,h\_\{\\setminus\}\\,\],\\qquad W\_\{g\}\\in\\mathbb\{R\}^\{g\\times d\}\.\(6\)WithggGELU\\operatorname\{GELU\}units andmmoperator pairs, the input side is\(g\+2​m\)​d\(g\+2m\)dand the output sideH​d=\(g\+k​m\)​dHd=\(g\+km\)d, so parameter neutrality with ad𝑓𝑓d\_\{\\mathit\{ff\}\}GELU\\operatorname\{GELU\}FFN holds when

m=2​d𝑓𝑓−2​g2\+k\.m=\\frac\{2\\,d\_\{\\mathit\{ff\}\}\-2g\}\{2\+k\}\.\(7\)We parameterize the split by theGELU\\operatorname\{GELU\}budget fractionρ=g/d𝑓𝑓\\rho=g/d\_\{\\mathit\{ff\}\}; our main model usesρ=0\.75\\rho=0\.75\(aGELU\\operatorname\{GELU\}\-dominant3:13\{:\}1split\)\. Two further details make the partition fair\. The Boolean output columns ofWoW\_\{o\}are*zero\-initialized*\(re\-zeroed after standard initialization\), so the layer begins as a puregg\-wideGELU\\operatorname\{GELU\}FFN and the set\-operator block is wired in only if training finds it useful \(a function\-preserving initialization\)\. And because the bounded Boolean block has a capped output norm whileGELU\\operatorname\{GELU\}is unbounded, we apply a per\-block RMS normalization \(a single scalar gain per block, parameter\-negligible\) beforeWoW\_\{o\}so the two blocks enter the read\-out on weight\-decay\-fair footing\.

### 3\.5A property: graceful degeneration

The sigmoid bounding endowsNC\-FFNwith a property its bilinear cousin lacks\. If the second operand of a unit becomes uninformative—Wb→0W\_\{b\}\\\!\\to\\\!0, henceB=σ​\(Wb​x\)→12B=\\sigma\(W\_\{b\}x\)\\\!\\to\\\!\\tfrac\{1\}\{2\}, a*non\-zero*constant—thenh∩,h∖→12​Ah\_\{\\cap\},h\_\{\\setminus\}\\\!\\to\\\!\\tfrac\{1\}\{2\}A, and the unit collapses to a single\-operand, sigmoid\-gated feature: an ordinary bounded activation\. The Boolean unit can therefore*relax*into a normal activation MLP wherever two\-operand logic is not useful\. A raw bilinear unit\(W1​x\)​\(W2​x\)\(W\_\{1\}x\)\(W\_\{2\}x\)has no such fallback: a linear projection has no non\-zero constant state, so zeroing one operand kills the unit, and its only single\-operand mode is the unbounded quadratic\(w​x\)2\(wx\)^\{2\}obtained whenW1∥W2W\_\{1\}\\\!\\parallel\\\!W\_\{2\}\. This difference—a graceful single\-operand fallback present under sigmoid bounding and absent for raw bilinear—is central to interpreting the depth\-resolved behavior we report in Section[7](https://arxiv.org/html/2606.31845#S7), and motivates the bilinear control of Section[4](https://arxiv.org/html/2606.31845#S4)\.

## 4Experimental Setup

### 4\.1Models

Our primary model is a 12\-layer, GPT\-2\-small\-scale decoder \(d=768d=768, 12 heads,d𝑓𝑓=3072d\_\{\\mathit\{ff\}\}=3072, sequence length20482048,125\.1125\.1M parameters\)\. TheNC\-FFNmodel replaces every FFN with the hybrid layer of Section[3\.4](https://arxiv.org/html/2606.31845#S3.SS4)atρ=0\.75\\rho=0\.75and operators\{∩,∖\}\\\{\\cap,\\setminus\\\}\(intersection and set\-difference\), giving, per layer,g=2304g=2304GELU\\operatorname\{GELU\}units andm=384m=384operator pairs \(768768Boolean hidden units\)\. The parameter\-matched baseline \(gelu\) is an identical architecture with standardGELU\\operatorname\{GELU\}FFNs\. We additionally use a 4\-layer variant of each architecture for cheaper ablations\.

### 4\.2Training

All models are trained on OpenWebText with the same recipe: batch size1616, sequence length20482048, AdamW \(β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay0\.10\.1\), peak learning rate3×10−43\\times 10^\{\-4\}with20002000warmup steps and cosine decay to10%10\\%of peak over one epoch \(272,687272\{,\}687steps\)\. We checkpoint weights at5050k\-step intervals \(5050k–250250k\) in addition to epoch boundaries, which enables the training\-trajectory analyses in Section[7](https://arxiv.org/html/2606.31845#S7)\. A second epoch \(analyzed in later sections\) continues from the epoch\-1 optimizer state with a fresh cyclical\-cosine cycle \(warmup500500, same peak and decay\), matching the multi\-epoch schedule of the baseline\.

### 4\.3Evaluation

We report dev perplexity and zero\-shot accuracy on a standard small\-model suite via thelm\-evalharness\(Gaoet al\.,[2021](https://arxiv.org/html/2606.31845#bib.bib30)\): LAMBADA\(Papernoet al\.,[2016](https://arxiv.org/html/2606.31845#bib.bib26)\), BLiMP \(67 subtasks\)\(Warstadtet al\.,[2020](https://arxiv.org/html/2606.31845#bib.bib25)\), ARC\-Easy\(Clarket al\.,[2018](https://arxiv.org/html/2606.31845#bib.bib29)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2606.31845#bib.bib27)\), and WinoGrande\(Sakaguchiet al\.,[2020](https://arxiv.org/html/2606.31845#bib.bib28)\)\. At this scale ARC\-Easy, HellaSwag, and WinoGrande carry little discriminating signal for all models—HellaSwag and WinoGrande sit at or near the random baseline—so we treat LAMBADA \(long\-range semantic prediction\) and BLiMP \(grammatical structure, with per\-subtask resolution\) as the discriminating benchmarks and report the rest for completeness\. All evaluations use a right\-padded batching adapter; we verified the adapter against published GPT\-2 LAMBADA accuracy\.

### 4\.4Probing the Boolean block

We use three complementary, inference\-only probes, all run on CPU from saved checkpoints\.

#### Engagement and realization\.

For each layer we capture the FFN input on held\-out tokens and recompute the operand activationsA,BA,Band the two block contributions to the residual write through the sharedWoW\_\{o\}\. We report \(i\) the*bool share*—the fraction of the per\-token FFN write carried by the Boolean block—as a measure of how much the model*uses*the operators, and \(ii\) operand statistics \(A¯\\bar\{A\},B¯\\bar\{B\}, andA⋅B¯\\overline\{A\\\!\\cdot\\\!B\}\) as a measure of whether genuine*two\-operand*logic is*realized*: a unit doing real set logic must have both an informativeAAand an informativeBB, whereas a unit withB≈0\.5B\\\!\\approx\\\!0\.5has degenerated to single\-operand gating \(Section[3\.5](https://arxiv.org/html/2606.31845#S3.SS5)\)\.

#### Causal ablation\.

To measure necessity rather than mere activation, we zero the Boolean output columns ofWoW\_\{o\}\(globally, or one layer at a time\) and measure the perplexity penalty on a fixed, paired held\-out batch\. Because a large ablation grows more damaging over training simply through co\-adaptation, we include a*control*: ablating an equal\-sized random subset of theGELU\\operatorname\{GELU\}columns, which isolates whether a growing Boolean ablation penalty is Boolean\-specific or generic\.

#### Logic\-unit scout\.

To find units doing genuine two\-operand logic and read their predicates, we rank unit pairs by two criteria:AAis a*selective*sparse gate \(low firing rate\) andBBis*informative whenAAfires*\(it takes both high and low values onAA\-firing tokens, rather than a constant\)\. For the resulting units we read the predicate by collecting the tokens that trigger theAA\-gate and then, amongAA\-firing positions, the tokens thatBB*includes*\(A⋅BA\\\!\\cdot\\\!B\) versus*excludes*\(A⋅\(1−B\)A\\\!\\cdot\\\!\(1\-B\)\)—rendering the unit’sand/and\-notstatement directly\.

### 4\.5The bilinear control ladder

Because our central claim is about the*view*—fuzzy set operators as an FFN—rather than a new operator, the key control is whetherNC\-FFNbehaves and explains differently from a matched bilinear FFN, the nearest prior art\(Pearceet al\.,[2025](https://arxiv.org/html/2606.31845#bib.bib5)\)\. A bilinear*layer*is itself competitive withGELU\\operatorname\{GELU\}\(Shazeer,[2020](https://arxiv.org/html/2606.31845#bib.bib4); Pearceet al\.,[2025](https://arxiv.org/html/2606.31845#bib.bib5)\), so it is a fair baseline rather than a weakened one, and a measured difference is attributable to what we change\. We therefore compare a ladder of parameter\-neutralρ=0\.75\\rho=0\.75hybrids that differ*only*in the non\-GELU\\operatorname\{GELU\}block: \(i\)NC\-FFN\[A⋅B,A⋅\(1−B\)\]\[A\\\!\\cdot\\\!B,\\,A\\\!\\cdot\\\!\(1\-B\)\]\(bounding\+\+complement\); \(ii\) a*sigmoid\-bilinear*blockσ​\(W1​x\)⋅σ​\(W2​x\)\\sigma\(W\_\{1\}x\)\\\!\\cdot\\\!\\sigma\(W\_\{2\}x\)\(bounding, no complement\), which—unlike raw bilinear—sharesNC\-FFN’s graceful degeneration and so isolates the contribution of the complement/set\-operator structure; \(iii\) a*raw bilinear*block\(W1​x\)⋅\(W2​x\)\(W\_\{1\}x\)\\\!\\cdot\\\!\(W\_\{2\}x\)\(neither\), which isolates the contribution of sigmoid bounding; and \(iv\) theGELU\\operatorname\{GELU\}anchor\. We realize this ladder in pure form as the four FFN types of the capability probe \(Section[5](https://arxiv.org/html/2606.31845#S5)\), and the same parameter\-neutral partition can be instantiated at language\-model scale as a control\.

## 5Results I: Reasoning Capability

Perplexity cannot answer our motivating question—whether a more structured FFN reasons more compactly—because next\-token prediction rewards distributional fit, not reasoning, and a model can be fluent without composing anything\. We therefore isolate capability with a controlled probe and measure it as a number\.

### 5\.1A capability probe: parity in pure feed\-forward stacks

NN\-bit parity is the canonical separator of additive from multiplicative computation\. In the±1\\pm 1encoding it is a single product∏ixi\\prod\_\{i\}x\_\{i\}, trivial for a multiplicative unit to*represent*but requiring exponentially many threshold units—and offering no benefit from depth—for a purely additive \(e\.g\. ReLU/GELU\) one\. It is exactly the kind of compositional function a structured component should handle more cheaply, and it is uncontaminated by the distributional statistics that let language models fake competence\.

To attribute any difference to the FFN alone we remove attention entirely: each model is a pure feed\-forward stack—a linear embedding of theNNinput bits \(→ℝ128\\to\\mathbb\{R\}^\{128\}\),LLresidual FFN blocks, and a linear head—trained directly to predict the parity bit\. We sweep the FFN*type*, the depthL∈\{1,2,3,4\}L\\in\\\{1,2,3,4\\\}, the hidden widthd𝑓𝑓d\_\{\\mathit\{ff\}\}, and the difficultyN∈\{1,…,12\}N\\in\\\{1,\\dots,12\\\}, with multiple seeds, and summarize each configuration by its*reach*: the largestNNit solves \(best accuracy≥0\.75\\geq 0\.75, averaged over seeds\)\. Four FFN types isolate the two properties our architecture toggles, here in*pure*form \(no GELU partition, so the block is entirely the named operator\):gelu\(additive\);*raw\-bilinear*\(W1​x\)⋅\(W2​x\)\(W\_\{1\}x\)\\\!\\cdot\\\!\(W\_\{2\}x\)\(multiplicative, unbounded\);*sigmoid\-bilinear*σ​\(W1​x\)⋅σ​\(W2​x\)\\sigma\(W\_\{1\}x\)\\\!\\cdot\\\!\\sigma\(W\_\{2\}x\)\(multiplicative, bounded, no complement\); andNC\-FFN\[σ​\(Wa​x\)​σ​\(Wb​x\),σ​\(Wa​x\)​\(1−σ​\(Wb​x\)\)\]\[\\sigma\(W\_\{a\}x\)\\sigma\(W\_\{b\}x\),\\,\\sigma\(W\_\{a\}x\)\(1\-\\sigma\(W\_\{b\}x\)\)\]\(multiplicative, bounded, with complement\)\. The networks are deliberately small: they are a measuring instrument for per\-parameter reasoning efficiency, not a scale claim\.

### 5\.2Bounding and depth control two different axes of reasoning

Figure[1](https://arxiv.org/html/2606.31845#S5.F1)and Table[1](https://arxiv.org/html/2606.31845#S5.T1)report reach as a function of depth at fixed width\. The picture is a clean dissociation rather than a single ordering\.

Table 1:Reasoning reach \(largestNN\-bit parity solved, mean over 5 seeds\) versus depthLL, pure feed\-forward stacks atd𝑓𝑓=256d\_\{\\mathit\{ff\}\}=256\. Bounded multiplicative units \(NC\-FFN, sigmoid\-bilinear\) dominate at shallow depth; unbounded raw\-bilinear is useless shallow but climbs steeply with depth; GELU plateaus between\.![Refer to caption](https://arxiv.org/html/2606.31845v1/x1.png)Figure 1:Reasoning reach versus depth for the four pure FFN types\. The two*bounded*multiplicative lines \(NC\-FFN, sigmoid\-bilinear\) start high and stay high;*unbounded*raw\-bilinear climbs steeply from a useless\-shallow base, tracking the degree\-doubling of composed products; the additive GELU baseline plateaus in between\. Bounding buys width\-efficient shallow reasoning; unbounding buys depth\-composition\.#### Bounding buys width\-efficient, shallow reasoning\.

At a single layer, the bounded multiplicative units are the most parameter\-efficient reasoning basis by a wide margin:NC\-FFNreachesN=8N\{=\}8and sigmoid\-bilinearN=7N\{=\}7, against GELU’s66and raw\-bilinear’s22\. A degree\-2 layer cannot solve parity\-8 by algebraic composition; the bounded units do it through*width*—combining sigmoid\-bounded product terms in the read\-out—and strikingly few suffice\. Sweeping the width at fixedL=1L\{=\}1\(Table[2](https://arxiv.org/html/2606.31845#S5.T2)\), each arm sits at a flat,*width\-robust*shallow ceiling that is already reached byd𝑓𝑓=16d\_\{\\mathit\{ff\}\}\{=\}16and unchanged out to256256:NC\-FFN88, sigmoid\-bilinear77, GELU66, and raw\-bilinear below55at every width\. This is exactly where bounding matters: products of\[0,1\]\[0,1\]values remain in\[0,1\]\[0,1\], so the read\-out can stably combine them, whereas an unbounded raw product explodes under summation—which is why raw\-bilinear is useless at one layer*at any width*\. Bounding is thus not a legibility nicety; it is what makes a multiplicative unit a usable shallow reasoning basis\.

Table 2:Reach versus hidden width at fixed depthL=1L\{=\}1\(mean over 4 seeds\)\. The shallow ordering is width\-robust—each arm sits at a flat ceiling fromd𝑓𝑓=16d\_\{\\mathit\{ff\}\}\{=\}16to256256\. Raw\-bilinear does not reachN=5N\{=\}5at any width \(it solves onlyN≤2N\\\!\\leq\\\!2\)\.
#### Unbounding buys depth\-composition\.

Raw\-bilinear shows the opposite profile: its reach climbs2→4→7→82\\\!\\to\\\!4\\\!\\to\\\!7\\\!\\to\\\!8with depth, tracking the algebraic fact that composing products doubles their degree per layer \(2L2^\{L\}\)\. Unbounded products cannot be summed in width but*can*be multiplied in depth, so depth is their only route to parity—and a clean one\. The bounded units do not gain comparably from depth \(they are already near ceiling atL=1L\{=\}1, and the sigmoid squashes degree growth across layers\), so the two strategies are genuinely complementary: bounding trades depth\-composition for width\-stability\.

#### The additive baseline plateaus\.

GELU sits between the two at every depth \(66–77\), neither width\-efficient nor depth\-composing for parity—consistent with parity being additive\-hard at any depth\. Its near\-flatness acrossLLconfirms that, unlike the multiplicative units, it derives no compositional benefit from added layers here\.

### 5\.3The complement adds a shallow margin

The two bounded arms differ only in the explicit complement, andNC\-FFNedges sigmoid\-bilinear where depth cannot be leaned on: atL=1L\{=\}1, reach88versus77\. With the operand budget tightest—one layer—getting bothA⋅BA\\\!\\cdot\\\!BandA⋅\(1−B\)A\\\!\\cdot\\\!\(1\-B\)from a single shared operand pair, rather than spending a separate pair to recover the complement by sign\-flip, is worth a parity bit\. The margin is modest and concentrated at shallow depth; we do not claim the complement adds reasoning power beyond the bounded multiplicative basis it shares with sigmoid\-bilinear—its larger contribution is to legibility \(Section[7](https://arxiv.org/html/2606.31845#S7)\)\. The honest summary is that*multiplicativity*and*bounding*are the load\-bearing capability properties, and the explicit negation is a small efficiency\-and\-legibility addition on top\.

#### Scope\.

These are controlled measurements on a single synthetic family with a fixed optimization budget; the absolute reach values depend on width, steps, and seeds, and a longer budget would likely let raw\-bilinear’s degree\-2L2^\{L\}advantage carry it higher atL=4L\{=\}4\. What is robust, and what we rely on, is the*qualitative dissociation*—bounded\-shallow versus unbounded\-deep, with the additive baseline between—which holds across the sweep\.

## 6Results II: Language Modeling

We now placeNC\-FFNin its target setting—a 125M\-parameter decoder trained on OpenWebText—and ask the necessary viability question: is an explicit\-combination FFN a faithful parameter\-neutral drop\-in, and what does the constraint cost? The answer is a clean tie on perplexity with a small, well\-localized cost on grammar, and—importantly—no language\-model*benefit*, exactly as the capability result predicts\.

### 6\.1A perplexity tie that tightens, and no semantic gain

At equal parameters,NC\-FFNmatches the GELU baseline on perplexity, and the match tightens with training \(Table[3](https://arxiv.org/html/2606.31845#S6.T3)\): the dev\-perplexity gap falls from\+0\.11\+0\.11after one epoch to\+0\.04\+0\.04after two, as the deep Boolean blocks relax toward GELU\-equivalent capacity \(Section[7](https://arxiv.org/html/2606.31845#S7)\)\. On the discriminating downstream benchmarks the model is at parity or slightly behind\. A single\-seed epoch\-1 LAMBADA edge \(accuracy\+0\.009\+0\.009, perplexity−7\-7\) does*not*survive a second epoch—by epoch two the baseline is marginally ahead on both—so we read LAMBADA as a tie and make no semantic\-advantage claim\. The floor benchmarks \(HellaSwag, WinoGrande\) are at chance for both models at this scale\.

Table 3:Language\-model evaluation, 12L, parameter\-matched, across two epochs\. The perplexity tie*tightens*with training; LAMBADA is a tie \(the epoch\-1 edge washes out\); the BLiMP deficit*persists and widens*\.
### 6\.2A small, persistent grammatical deficit—and where it lives

The one robust downstream signal is a BLiMP deficit, and it*widens*with training rather than closing: the baseline’s grammar improves over the second epoch \(0\.807→0\.8230\.807\\\!\\to\\\!0\.823\) whileNC\-FFN’s stays flat \(0\.788→0\.7900\.788\\\!\\to\\\!0\.790\), so the gap grows from−0\.019\-0\.019to−0\.033\-0\.033\(well outside within\-sample error\)\. The deficit is concentrated, not diffuse: across the 67 subtasks the baseline wins 36 to 22 \(9 ties\), with losses clustered in*island/long\-distance extraction*\(left\-branch\-island−0\.31\-0\.31,coordinate\-structure−0\.25\-0\.25\),*NPI licensing*\(npi\-present\-1−0\.18\-0\.18,matrix\-question\-npi−0\.14\-0\.14\), and*quantifiers*\(−0\.10\-0\.10to−0\.13\-0\.13\)\. There is an irony worth stating plainly: NPI licensing—the grammatical reflex of negation and polarity—is among the negation\-motivated architecture’s*weakest*areas \(lone exception:only\-npi\-licensor\-present, its largest gain at\+0\.23\+0\.23\)\. The architectural negation primitive does not translate into better handling of grammatical negation\.

### 6\.3Why: capacity allocation, not missing capability

We attribute the deficit to capacity*allocation*rather than an inability\. Per parameter, the hidden units cost the same as GELU’s \(2​d2deach\), but a Boolean unit is a*more constrained*basis\. A pair’s two outputs satisfyA⋅B\+A⋅\(1−B\)=AA\\\!\\cdot\\\!B\+A\\\!\\cdot\\\!\(1\-B\)=A, so the pair spans only\{A,A⋅B\}\\\{A,\\,A\\\!\\cdot\\\!B\\\}: one standalone sigmoid feature and one product, with the second operand spent entirely inside the product\. Two GELU units of the same parameter cost instead give two*independent*standalone features\. Ordinary language modeling is hungry for standalone, threshold\-like features \(which is what grammar needs\), and mostly does not need the products—so a quarter ofNC\-FFN’s budget is, for this task, spent on units of lower marginal value\. Its effective standalone\-feature capacity is accordingly between that of a strictly narrower GELU model and the full baseline, which is the deficit’s size and sign\. This is the same allocation that the capability probe rewards: the products that cost grammar a little are exactly what reason on parity\. The architecture’s capability is real but*dormant*for next\-token prediction, which does not call for it—so it ties, at a small constant cost, rather than winning or losing decisively\.

## 7Results III: Legibility and Its Dynamics

Because everyNC\-FFNunit is a namedand/and\-notof two operands, the layer admits questions an opaque activation does not:*where*in the network is genuine two\-operand logic realized,*which*units are causal, and*how*does the logical content change over training? We answer these with inference\-only probes \(Section[4](https://arxiv.org/html/2606.31845#S4)\) and close with the result we find most telling—that the amount of logic is not a property of the architecture but of what the task rewards\.

### 7\.1Engagement is depth\-uniform; realization is localized to layer 0

The Boolean block is heavily and uniformly*used*: its share of the per\-token FFN residual write is0\.300\.30–0\.470\.47at every layer, with no decay toward the top \(Table[4](https://arxiv.org/html/2606.31845#S7.T4)\)\. But genuine*two\-operand*realization is sharply localized\. Only at layer 0 is the second operand informative \(B¯=0\.48\\bar\{B\}=0\.48, away from the uninformative0\.50\.5\) with a sizable intersection signal \(A⋅B¯=0\.13\\overline\{A\\\!\\cdot\\\!B\}=0\.13\); at every deeper layerB¯\\bar\{B\}collapses to≈0\.50\\approx 0\.50andA⋅B¯\\overline\{A\\\!\\cdot\\\!B\}drops by an order of magnitude, so the units have degenerated to the single\-operand gates of Section[3\.5](https://arxiv.org/html/2606.31845#S3.SS5)\. We summarize this as a*recruitment\-versus\-realization*gap: the partition is recruited everywhere but its set\-operator semantics are realized only at the embedding\-adjacent layer—a contrarian result, cutting against the intuition that depth sharpens abstraction\.

Table 4:Boolean\-block statistics by layer \(final checkpoint\)\.*bool share*: fraction of the FFN residual write carried by the Boolean block \(engagement, depth\-uniform\)\.A⋅B¯\\overline\{A\\\!\\cdot\\\!B\},A¯\\bar\{A\},B¯\\bar\{B\}: operand realization—only layer 0 shows informative two\-operand structure\.
### 7\.2The split is used or cleanly retired, almost never redundant

A split representation invites a specific failure—the two operands collapsing into a*redundant*copy, so the layer pays for a duplicate\. Section[3\.5](https://arxiv.org/html/2606.31845#S3.SS5)predicts a different failure mode instead: an unused unit drops*one*operand and reverts to an ordinary single\-operand activation\. We test this directly, classifying every Boolean unit from the joint statistics of its operands across tokens\. An operand is*collapsed*when its activation variance is negligible; a surviving pair is*redundant*whenBBcorrelates withAAabove the9999th percentile of a matched cross\-unit floor, and*independent*otherwise\. The three fates are exhaustive and consistent across the quantifier ladder \(Table[5](https://arxiv.org/html/2606.31845#S7.T5)\)\.*Redundant*duplicate pairs areunder5%5\\%in every variant, with median operand correlation≈0\\approx 0\(frequently negative—the “AAand notBB” regime\)\. The remainder divides into units that retire one operand to the single\-operand gate of Section[3\.5](https://arxiv.org/html/2606.31845#S3.SS5)\(≈45%\\approx\\\!45\\%\) and units whose operands stay both active and independent \(≈50%\\approx\\\!50\\%\)\. The split therefore does not waste capacity on a duplicate: where two\-operand logic is not used, a unit relaxes to a plain activation, exactly the graceful degeneration the bounding affords\. We stress that “independent” is a far weaker condition than the*informative*two\-operand signal of Table[4](https://arxiv.org/html/2606.31845#S7.T4)\(concentrated at layer 0\) or the crisp predicates of Table[7](https://arxiv.org/html/2606.31845#S7.T7): an operand may vary around its uninformative midpoint, contributing distinct but small\-magnitude modulation\. The only claim here is the one the architecture promises—that the second operand, when present, is not a redundant copy of the first\.

Table 5:Operand fate over all Boolean units \(epoch\-one checkpoints\), averaged across the1212layers\.*One\-operand*: one operand collapsed to≈\\approxconstant \(the single\-operand relaxation of Section[3\.5](https://arxiv.org/html/2606.31845#S3.SS5)\)\.*Independent*: both operands active, correlation at/below a matched cross\-unit floor \(distinct, non\-duplicated signal\)\.*Redundant*: both active but the second tracks the first above the floor\. The split is almost never a redundant copy\.
### 7\.3Causal ablation: layer\-0 is irreplaceable, deep blocks are redundant capacity

Activation statistics measure use, not necessity\. Zeroing the Boolean output columns and measuring the perplexity penalty on a fixed held\-out batch gives the causal picture \(Table[6](https://arxiv.org/html/2606.31845#S7.T6)\)\. Removing*only layer 0*’s Boolean block costs≈187\\approx 187perplexity \(\+2\.23\+2\.23nats\)—the single most load\-bearing FFN component in the network, and the one layer doing genuine two\-operand logic\. Deeper layers are individually near\-removable \(11–66ppl each\) but*collectively*essential: ablating all Boolean blocks costs\+5\.16\+5\.16nats, well above the≈3\.5\\approx 3\.5nats of the summed single\-layer penalties—a distributed, redundant Boolean pathway across depth\. A matched control \(an equal\-sized randomGELU\\operatorname\{GELU\}\-column ablation\) grows nearly as steeply over training \(\+1\.11→\+4\.63\+1\.11\\\!\\to\\\!\+4\.63\), so most of the all\-block growth is generic co\-adaptation, not Boolean necessity; what survives the control is that Boolean columns are more load\-bearing thanGELU\\operatorname\{GELU\}columns at every checkpoint, an advantage that narrows by the end—the deep blocks converging toward generic capacity, the localized layer\-0 block remaining decisive\.

Table 6:Causal ablation: loss penalty \(nats\) for zeroing Boolean output columns, paired on a fixed131131k\-token batch\. TheGELU\\operatorname\{GELU\}\-chunk control ablates an equal number of randomGELU\\operatorname\{GELU\}columns/layer\.
### 7\.4Readable units, and a logic–fuzzy spectrum

Requiring a selectiveAA\-gate and a genuinely two\-valuedBByields a small set—fewer than ten strict two\-operand units in the whole 12\-layer model, almost all at layer 0—that read as recognizable predicates*without*dictionary learning\. The clearest \(layer\-0,AA\-firing rate0\.0850\.085\) is a function/content discriminator: itsAA\-gate fires for high\-frequency function tokens \(“,”, “\\n”, “the”, “\.”\); theA∩BA\\\!\\cap\\\!Bbranch keeps the function tokens and theA∖BA\\\!\\setminus\\\!Bbranch isolates content tokens \(“Antonio”, “Nintendo”\)—a clean “frequent tokenand notcontent word\.” Others encode sentence/line\-boundary context and a code\-syntax\-versus\-identifier split\. But this is a*spectrum*, not a dichotomy \(Table[7](https://arxiv.org/html/2606.31845#S7.T7)\): of 257 active layer\-0 units, 221 have negligibleBB\-discrimination \(soft single\-operand gating\), with a thin tail of crisp logic; a deep layer has no tail at all\. The directly\-readable units are few and encode low\-level structural distinctions; the bulk of operands are distributed and polysemantic, exactly as superposition predicts\(Elhageet al\.,[2022b](https://arxiv.org/html/2606.31845#bib.bib20)\)\. The architectural payoff is not universal monosemanticity but that*even the distributed units carry explicit logical form*—each a knownand/and\-not—whereas aGELU\\operatorname\{GELU\}or bilinear ensemble is distributed*and*opaque in its operation\.

Table 7:The logic–fuzzy spectrum at layer 0\. Units binned byBB\-discrimination whenAAfires \(≈0\\approx 0= soft single\-operand gating\)\. A deep layer \(6\) has no logic tail\.
### 7\.5Redundant units, causally\-used features

Are the readable units*causal*? Single\-unit interventions say no: ablating the function/content unit’s readout, or flipping itsBBoperand, changes next\-token predictions negligibly \(<10−3<10^\{\-3\}nats\)\. This is redundancy, not irrelevance: training distributes each feature across many carriers, so no single unit is necessary \(consistent with the super\-additive block ablation above\)\. The correct test is at the level of*features*: ablating a feature*direction*removes it from all carriers at once\. Projecting the function/content axis out of the layer\-0 output costs148×148\\timesthe loss of a random direction of equal norm, localized to function\-word positions \(1\.6×1\.6\\times\); the sentence\-boundary unit’s own axis gives228×228\\timesa random direction and4\.2×4\.2\\timeslocalization\. The faithful statement is therefore not “this unit caused tokenXX” but“a unit’s firing is faithful evidence that a causally\-used feature is active here”—and, methodologically, that under a redundant network one must ablate*directions*, not units, or systematically understate importance\.

### 7\.6The logic is task\-shaped: it hardens where the task pays, erodes where it does not

The most revealing property is dynamic\. The layer\-0 two\-operand signal does not sharpen with training; it*softens*—A⋅B¯\\overline\{A\\\!\\cdot\\\!B\}decays monotonically across both epochs \(0\.195→0\.150→0\.134→0\.111→0\.096→0\.0870\.195\\\!\\to\\\!0\.150\\\!\\to\\\!0\.134\\\!\\to\\\!0\.111\\\!\\to\\\!0\.096\\\!\\to\\\!0\.087\), as the products relax toward the single\-operand gating that language modeling rewards \(Section[6](https://arxiv.org/html/2606.31845#S6)\)\. Crisp two\-operand logic is an*early*\-training feature that erodes; what persists is the depth\-localization, the engagement, and feature\-level causality \(the sentence\-boundary axis is still442×442\\timesa random direction at epoch two\), not unit\-level crispness\.

Is the erosion intrinsic, or a readout of the objective? We test the converse directly\. Train the*same*Boolean block, with the same probes, on a task that*rewards*products—parity at the solvable edge—and track the logic over training\. It does the opposite: the Boolean structure*hardens*, and the hardening coincides exactly with generalization\. Across 15 runs \(N∈\{7,8,9\}N\\in\\\{7,8,9\\\}, 5 seeds\), grok\-aligning each trajectory to its own accuracy jump, the fraction of units doing genuine two\-operand logic goes from0\.0060\.006*before*the grok step to0\.4750\.475*at*it—a step change in logical content locked to the moment the network acquires the capability \(Figure[2](https://arxiv.org/html/2606.31845#S7.F2), left\)\. The contrast with the language\-model erosion \(right\) is the result: same architecture, same measurement, opposite trajectories\. The network keeps exactly the logic its objective pays for—crystallizing it at the instant of reasoning, relaxing it away where only prediction is rewarded\. “How much logic” is thus not a fixed attribute of the layer but a legible readout of what the task demands of it\.

![Refer to caption](https://arxiv.org/html/2606.31845v1/x2.png)

![Refer to caption](https://arxiv.org/html/2606.31845v1/x3.png)

Figure 2:The logical content is task\-shaped\.Left:on a product\-rewarding task \(parity\), the fraction of units doing two\-operand logic snaps from≈0\\approx 0to≈0\.48\\approx 0\.48*at*the grok step \(grok\-aligned mean over 15 runs, band==s\.d\.\)\.Right:under language\-model training, the layer\-0 two\-operand signal*erodes*monotonically across two epochs\. Same Boolean block, same probe, opposite trajectories\.

## 8Results IV: A Trainability Ceiling at High Boolean Fraction

The architecture of Section[5](https://arxiv.org/html/2606.31845#S5)is a partition: a fraction of each FFN is the GELU baseline and the remainder is the Boolean block, with the paper’s configuration using a25%25\\%Boolean share\. A natural question for an*end\-to\-end*\-legible model is how far that share can be pushed— ideally to100%100\\%, a purely Boolean FFN with no GELU at all\. We find a sharp obstacle that is not about accuracy but about*optimization*: as the Boolean fraction rises, the model trains normally for a long time and then diverges abruptly, and the more Boolean the FFN, the sooner it does so\.

### 8\.1The trainable horizon shrinks as the Boolean fraction grows

We trained the1212\-layer,125125M\-parameter model for one epoch on OpenWebText while sweeping the Boolean fraction \(all layers,\{∩,∖\}\\\{\\cap,\\setminus\\\}operators, parameter\-matched throughout\), and recorded whether each run completed the epoch or diverged\. We declare divergence when the*instantaneous*perplexity exceeds100100after a1515k\-step grace window, at which point training is halted automatically and a restartable checkpoint is saved\. Table[8](https://arxiv.org/html/2606.31845#S8.T8)reports the outcome\.

Table 8:Trainability versus Boolean fraction \(1212L, one epoch, all layers\{∩,∖\}\\\{\\cap,\\setminus\\\}, parameter\-matched\)\. Below50%50\\%the model completes the epoch; above it, every run diverges, and the divergence step falls as the fraction rises\. Single seed per configuration; the90%90\\%/95%95\\%ordering is within run\-to\-run noise\.The picture is a*trainable\-horizon*effect rather than a clean threshold: a25%25\\%or50%50\\%Boolean model trains to convergence \(the50%50\\%model reaching a dev perplexity of21\.621\.6versus the GELU baseline’s19\.019\.0\), whereas every model above50%50\\%eventually diverges, and the step at which it does so grows shorter as the Boolean share grows—from∼\\sim136136k steps at75%75\\%down to roughly sixteen thousand for the pure model\. Crucially, the instability is also*latent*: the75%75\\%model trained healthily to step123123k \(perplexity≈38\\approx 38, indistinguishable from a stable run\) before diverging at136136k, so a run that “looks fine” early is not yet safe\. Even a single Boolean\-bearing layer in an otherwise\-GELU stack is enough to seed the failure, though it postpones it dramatically \(a model with the Boolean block in layer0only diverged at step202202k\)\.

### 8\.2The failure is a sudden spike, invisible to the running average

The divergence is not a gradual drift but a near\-instantaneous event\. Across the75%75\\%model’s blow\-up, the*cumulative*epoch perplexity moved only from9292to9494while the*instantaneous*perplexity jumped from5151to305305within a single250250\-step logging window\. The running average—the quantity ordinarily monitored during training—barely registers the event until it is far advanced, which is why our divergence criterion and the early\-stopping guard both operate on instantaneous, not cumulative, loss\. We note this as a small methodological point: characterizing this failure mode at all requires watching the instantaneous signal\.

### 8\.3It is not the magnitude of the residual write

A natural hypothesis is that the Boolean block’s contribution to the residual stream grows without bound and a single large write destabilizes the model\. The Boolean*features*are already bounded—they are sigmoid products in\[0,1\]\[0,1\]—so the only unbounded quantity is the read\-out matrixWoW\_\{o\}\. We tested two interventions that bound the residual write directly\. A parallel linear bypass added to a pure Boolean layer delayed its divergence by roughly four\-fold \(step16\.516\.5k to6969k\) but did not prevent it; normalizing the FFN output to unit RMS \(a fixed\-magnitude residual write regardless of‖Wo‖\\\|W\_\{o\}\\\|\) likewise only postponed the pure model’s divergence \(to step1919k\)\.*Neither bounding the write nor a stabilizing linear path removes the instability*, which argues against a write\-magnitude explanation and points instead to an ill\-conditioning of the optimization itself\.

A plausible—but unverified—mechanism is consistent with the interpretability measurements of Section[7](https://arxiv.org/html/2606.31845#S7)\. Training drives the operand projections well above their initialization scale \(the row norm ofWaW\_\{a\}grows two\- to four\-fold over training\) and pushes roughly half of the operand values into the saturated tails of the sigmoid\. In that regime the gradient of a bounded productσ​\(Wa​x\)​σ​\(Wb​x\)\\sigma\(W\_\{a\}x\)\\,\\sigma\(W\_\{b\}x\)is dominated by the small, sharply varying derivative of a saturated sigmoid, and a stack of such layers is prone to the kind of sudden, self\-reinforcing update we observe\. A sufficient minority of GELU units—the partition—supplies an unsaturating gradient path that keeps the optimization well\-conditioned, which is why a partition buys*horizon*\(more Boolean delays the onset\) without buying*immunity*\.

### 8\.4Implication

Legibility, in this architecture, is therefore free only up to a partition\. At the25%25\\%Boolean share we study, the model trains exactly as well as its GELU baseline; one can push to50%50\\%at a small perplexity cost; but a fully Boolean, end\-to\-end\-legible FFN is not trainable by the present recipe, and the two obvious fixes—bounding the write, adding a linear highway—only delay the failure\. The obstacle to an explanatory language model in which*every*feed\-forward unit carries explicit logical form is thus not representational but optimization\-theoretic: the high\-Boolean\-fraction ceiling is consistent with a saturated\-product gradient that bounding the residual write does not cure\. Removing it is the central open problem this work leaves behind\.

## 9Results V: Self\-Forgetting Quantifiers and Legible Licensing

Two results above boundNC\-FFN’s legibility, and they point the same way\. Section[7](https://arxiv.org/html/2606.31845#S7)found that genuine two\-operand logic is realized only at the embedding\-adjacent layer and*erodes*toward soft gating under language\-model training\. Section[6](https://arxiv.org/html/2606.31845#S6)traced the model’s one robust grammatical deficit to a specific cluster of constructions—NPI licensing, long\-distance extraction, and quantifiers—i\.e\. exactly the phenomena a set\-logic FFN ought to be good at\. Both diagnoses share a cause: the operators ofNC\-FFNare*within\-token*\. Intersection and set\-difference combine features at the current position; they say nothing about whether a feature has*already occurred*earlier in the sequence, which is what licensing and quantification require \(“there is a comparative*somewhere to the left*, so license*than*”\)\.

We close the paper by adding the missing primitive—a fuzzy*quantifier*over the sequence—and find that it converts the limitation into the work’s strongest legibility result\. A small block of*self\-forgetting*fuzzy quantifiers \(i\) recovers the grammatical deficit and modestly exceeds the baseline, \(ii\)*holds*its logical structure through training—and pushes it*into depth*rather than eroding—and \(iii\) reads, at the semantic layers, as a bank of grammatical*licensing detectors*: crisp, short\-lived, and individually nameable without dictionary learning\. The decay is the crux: it is what keeps the quantifier a*predictive, local*operator rather than a sticky latch, and what makes the mechanism fit how a transformer actually processes a sequence\.

### 9\.1A fuzzy\-quantifier block with learned forgetting

To the hybrid layer of Section[3\.4](https://arxiv.org/html/2606.31845#S3.SS4)\(theρ=0\.75\\rho=0\.75GELU\\operatorname\{GELU\}/Boolean partition\) we add a third, parameter\-neutral block ofmqm\_\{q\}*quantifier*units\. Each unit forms a per\-token fuzzy membership from the residual stream,

Mt=σ​\(Wq​xt\)∈\[0,1\]mq,Wq∈ℝmq×d,M\_\{t\}=\\sigma\(W\_\{q\}\\,x\_\{t\}\)\\in\[0,1\]^\{m\_\{q\}\},\\qquad W\_\{q\}\\in\\mathbb\{R\}^\{m\_\{q\}\\times d\},\(8\)and the membership is aggregated*along the sequence*by two causal fuzzy quantifiers, a soft existential and a soft proportion, each with its own*learned*forgetting rate:

\(“∃\\existsrecently”\)Et=max⁡\(Mt,γ⊙Et−1\),\\displaystyle E\_\{t\}=\\max\\\!\\big\(M\_\{t\},\\ \\gamma\\odot E\_\{t\-1\}\\big\),\(9\)\(“proportion recently”\)Pt=\(1−λ\)⊙Mt\+λ⊙Pt−1,\\displaystyle P\_\{t\}=\(1\-\\lambda\)\\odot M\_\{t\}\+\\lambda\\odot P\_\{t\-1\},\(10\)with per\-unit decaysγ=σ​\(θγ\)\\gamma=\\sigma\(\\theta\_\{\\gamma\}\)andλ=σ​\(θλ\)\\lambda=\\sigma\(\\theta\_\{\\lambda\}\)in\(0,1\)\(0,1\)\. The two limits are the non\-forgetting quantifiers:γ=1\\gamma=1makes \([9](https://arxiv.org/html/2606.31845#S9.E9)\) a cumulative maximum \(“MMfired at*some*earlier token”\), andλ=1\\lambda=1makes \([10](https://arxiv.org/html/2606.31845#S9.E10)\) a cumulative mean \(“the*fraction*of earlier tokens that fired”\)\. We*initialize*the decays near this non\-forgetting limit \(γ0,λ0≈0\.99\\gamma\_\{0\},\\lambda\_\{0\}\\\!\\approx\\\!0\.99\), so the block begins as a true sticky quantifier and must*learn*any forgetting it adopts\. The aggregated streamsEt,PtE\_\{t\},P\_\{t\}are written to the residual through the same shared,*zero\-initialized*read\-outWoW\_\{o\}as the Boolean block, so the layer again begins as a pureGELU\\operatorname\{GELU\}FFN and wires the quantifiers in only if useful\. We study three members of a ladder, each over the same\{∩,∖\}\\\{\\cap,\\setminus\\\}Boolean base:\+quant\(fixedγ=λ=1\\gamma\\\!=\\\!\\lambda\\\!=\\\!1, no forgetting\),\+decay\(the learnedγ,λ\\gamma,\\lambdaabove\), and\+decay\+gate\(additionally a per\-layer scalar gateβ=σ​\(θβ\)\\beta=\\sigma\(\\theta\_\{\\beta\}\)on the quantifier write, a learned per\-depth volume control\)\. Decay and gate parameters are per\-unit/per\-layer scalars and are parameter\-negligible;mqm\_\{q\}is chosen to hold the layer parameter\-neutral with the125125M baseline\. Logically, each unit is a*unary*predicateMMquantified over position by \([9](https://arxiv.org/html/2606.31845#S9.E9)\)–\([10](https://arxiv.org/html/2606.31845#S9.E10)\): the block adds*monadic*sequence quantification, notnn\-ary relations, and the licensor\-to\-licensee dependency it captures is a relation between*positions*bound by the learned decay rather than annn\-ary predicate \(we return to this scope in Section[10](https://arxiv.org/html/2606.31845#S10)\)\.

### 9\.2The quantifier recovers the grammatical deficit—and ties LAMBADA on top

The motivating symptom was grammatical:NC\-FFNlost−0\.019\-0\.019BLiMP to the baseline, concentrated in licensing \(Section[6](https://arxiv.org/html/2606.31845#S6)\)\. Adding the quantifier block*closes that gap at epoch 1*\(Table[9](https://arxiv.org/html/2606.31845#S9.T9)\)\.\+decay\+gaterecovers BLiMP to0\.8100\.810—level with the0\.8070\.807GELU\\operatorname\{GELU\}baseline and well aboveNC\-FFN’s0\.7880\.788—while simultaneously posting the strongest LAMBADA of any model here \(0\.2710\.271vs\. the baseline’s0\.2540\.254\), at unchanged parameters and on standard attention\. The recovery is*monotone in the ladder*: the non\-forgetting\+quantonly*partially*closes the gap \(0\.7950\.795, aboveNC\-FFN’s0\.7880\.788but short of baseline\),\+decayreaches0\.8000\.800, and\+decay\+gate0\.8100\.810—so it is the forgetting, not the bare aggregation, that closes the grammar\. Epoch 2 sharpens this: the baseline’s grammar pulls ahead \(0\.8230\.823\), but the quantifier*halves*NC\-FFN’s now\-wider deficit \(0\.8100\.810vs\.NC\-FFN’s0\.7900\.790\), and\+decayposts the best LAMBADA of any model here \(0\.2790\.279, edging the0\.2750\.275baseline—single seed, so we read it as a tie\-or\-slight\-lead\)\. The mechanism we add to*recover*the licensing deficit is the same one that, below, reads as licensing detectors, and it is the*decay*that makes the quantifier both grammatical and legible\.

Table 9:Adding a sequence quantifier toNC\-FFNcloses its grammatical deficit and leads the ladder on LAMBADA \(1212L, parameter\-matched, single seed, both epochs\)\. At epoch 1\+decay\+gateerasesNC\-FFN’s−0\.019\-0\.019BLiMP deficit \(0\.8100\.810, level with the0\.8070\.807baseline\) and posts the best LAMBADA; at epoch 2 the baseline’s grammar pulls ahead \(0\.8230\.823\) but the quantifier*halves*the now\-widerNC\-FFNdeficit, and\+decayposts the best LAMBADA of any model here \(0\.2790\.279\)\. The BLiMP recovery is monotone in the decay ladder \(\+\+quant<<\+\+decay<<\+\+decay\+\+gate\)\.We read this conservatively—single seed, though now consistent across both epochs—but the qualitative pattern is clean and is corroborated mechanistically below: the FFN block whose*job*is sequence quantification removes the deficit on the grammar that needs sequence quantification\. Confirming that the recovery is concentrated in the same licensing subtasksNC\-FFNlost \(rather than diffuse\) is the natural follow\-up; here we establish the aggregate recovery and the mechanism behind it\.

### 9\.3Now the structure holds—and migrates into depth

The most worrying property ofNC\-FFNwas dynamic: its two\-operand logic was an*early*\-training feature that eroded toward soft gating \(Section[7\.6](https://arxiv.org/html/2606.31845#S7.SS6)\)\. The quantifier model does not erode\. Re\-running the realization probe across the same training checkpoints \(50​k→250​k→50\\text\{k\}\\to 250\\text\{k\}\\toepoch\) tells the opposite story \(Table[10](https://arxiv.org/html/2606.31845#S9.T10)\)\. The load\-bearing intersection signalA⋅B¯\\overline\{A\\\!\\cdot\\\!B\}does not collapse to layer 0 and decay; it*redistributes toward depth*—falling at layer 0 \(0\.059→0\.0370\.059\\to 0\.037\) while the deep layers roughly*double*\(L8​0\.008→0\.014\\text\{L8\}\\;0\.008\\to 0\.014,L9​0\.011→0\.018\\text\{L9\}\\;0\.011\\to 0\.018\), with the layer\-average tracing a shallow U \(a mid\-training dip that recovers to its starting value by the epoch\)\. Two surface statistics do soften over training \(std​\(B\)​0\.26→0\.20\\mathrm\{std\}\(B\)\\;0\.26\\to 0\.20,crisp​\(A,B\)​0\.53→0\.48\\mathrm\{crisp\}\(A,B\)\\;0\.53\\to 0\.48\), but the operative quantity—realized two\-operand intersection—is held and*deepened*\. This is the reversal we were after: where the within\-token Boolean logic ofNC\-FFNdegenerated to anL​0\\text\{L\}0relic, the sequence\-quantifier model spreads genuine logic across the stack and keeps it there \(Figure[3](https://arxiv.org/html/2606.31845#S9.F3)\)\.

![Refer to caption](https://arxiv.org/html/2606.31845v1/boolean_over_training_AB.png)Figure 3:A⋅BA\\\!\\cdot\\\!Bintersection magnitude per layer across epoch\-1 training \(\+decay\+gate\)\. UnlikeNC\-FFN’sL​0\\text\{L\}0erosion, the intersection signal migrates out of layer 0 into the deep layers rather than collapsing—genuine two\-operand logic is held and pushed into depth\.The decay/gate split also separates here, and it is informative\.*Learned decay*preserves quantifier crispness best \(shallow layers even sharpen; only the top two or three soften, mean0\.858→0\.8320\.858\\to 0\.832\)\. The*gate*\(\+decay\+gate\) trades some of that away at the deepest layers \(0\.840→0\.7550\.840\\to 0\.755;L​11\\text\{L\}110\.83→0\.510\.83\\to 0\.51\) in exchange for its LAMBADA/BLiMP gains\. We accordingly treat\+decayas the clean mechanistic object and the gate as a performance\-oriented depth\-shaping refinement\.

Table 10:Realization migrates into depth \(\+decay\+gate,A⋅B¯\\overline\{A\\\!\\cdot\\\!B\}by layer over training\)\. Contrast Section[7](https://arxiv.org/html/2606.31845#S7), whereNC\-FFN’s two\-operand signal was anL​0\\text\{L\}0\-only feature that decayed: here it falls atL​0\\text\{L\}0and*rises*in the deep layers\.
### 9\.4The decay un\-learns the latch: forgetting, not memory

The quantifiers \([9](https://arxiv.org/html/2606.31845#S9.E9)\)–\([10](https://arxiv.org/html/2606.31845#S9.E10)\)*start*as sticky operators \(γ0≈0\.99\\gamma\_\{0\}\\\!\\approx\\\!0\.99: a cumulative max never forgets\)\. A model that simply rode that initialization would be implementing a permanent latch—“once true, true forever”—which is not how a language model processes a sequence and would make the operator a poor fit\. It does not\. Reading the learned decays off the trained weights and converting them to token half\-livest1/2=ln⁡\(0\.5\)/ln⁡γt\_\{1/2\}=\\ln\(0\.5\)/\\ln\\gammashows that*every*unit moved decisively away from the latch \(Table[11](https://arxiv.org/html/2606.31845#S9.T11)\):

- •No near\-permanent units\.Not one of the15361536existential units \(and none of the proportional units\) hasγ\>0\.97\\gamma\>0\.97in either learned model; the maximum isγ=0\.963\\gamma=0\.963, and only0\.07%0\.07\\%exceedγ=0\.90\\gamma=0\.90\. From aγ0≈0\.99\\gamma\_\{0\}\\\!\\approx\\\!0\.99start, the decays were*learned down*\.
- •Short, local memory\.The median existential half\-life is1\.521\.52tokens \(proportional1\.321\.32\);88\.5%88\.5\\%of existential units forget within two tokens\. The quantifier became a short recency detector, not a store\.
- •A mid\-stack lifetime hump\.Half\-life peaks at the middle layers \(L4–7,∼\\sim1\.71\.7tokens\) and is shortest at the edges \(L0, L8–11,∼\\sim1\.21\.2–1\.41\.4\); the per\-layer gateβ\\betais*highest*exactly where lifetimes are shortest \(peakβ=0\.80\\beta=0\.80at L9\), i\.e\. the model leans on the quantifier most for ephemeral, per\-token detection\.

That the operator un\-learns its own stickiness is the result that makes it transformer\-appropriate, and it sets up the reading of the next subsection: a∼\\sim1\.51\.5\-token memory is precisely long enough to carry a*prediction*from one token to the next, and no longer \(Figure[4](https://arxiv.org/html/2606.31845#S9.F4)\)\.

![Refer to caption](https://arxiv.org/html/2606.31845v1/decay_halflife_by_layer.png)Figure 4:Learned existential \(∃\\exists\) token half\-life per layer \(median∼\\sim1\.51\.5tokens\)\. From a stickyγ0≈0\.99\\gamma\_\{0\}\\\!\\approx\\\!0\.99initialization, every unit learned a short, local memory;*zero*units remain near\-permanent\.Table 11:Learned forgetting \(existential quantifier, both learned models\)\. Initialized at a stickyγ0≈0\.99\\gamma\_\{0\}\\\!\\approx\\\!0\.99\(cumulative max\), every unit learned a short half\-life;*zero*units remain near\-permanent\. Half\-lives in tokens,t1/2=ln⁡\(0\.5\)/ln⁡γt\_\{1/2\}=\\ln\(0\.5\)/\\ln\\gamma\.
### 9\.5The legible units are grammatical*licensing detectors*

At the semantic layers where the quantifier write is largest \(L9–L11, gateβ=0\.80/0\.75/0\.56\\beta=0\.80/0\.75/0\.56\), the membership units are individually nameable, and they realize a single, consistent computation:*predictive grammatical licensing*\. Crucially—and this is the cleanest part of the story—a unit does*not*fire on the token it decodes to\. Reading a unit’s write direction through the unembedding \(a logit lens over itsWoW\_\{o\}column\) names a*function word*; but the membershipM=σ​\(Wq​x\)M=\\sigma\(W\_\{q\}x\)itself fires one token*earlier*, on the grammatical*licensor*that predicts that function word, and the short\-memory existential then carries it forward to the position where the function word is emitted\. We verified the literal\-token reading is false: for the comparative unit, mean membership on ‘ than’ is0\.050\.05, but on its licensors ‘ more’/‘ less’ it is0\.550\.55–0\.640\.64\. Four such detectors recur across both learned models \(Table[12](https://arxiv.org/html/2606.31845#S9.T12)\):

Table 12:Four grammatical*licensing detectors*\(layer 9, recurring across the\+decayand\+decay\+gatemodels\)\. The membership fires on the*licensor*; the unit writes toward the*licensed*function word it predicts; the∼\\sim1\.51\.5\-token decay bridges the gap\. None fires on the function word it names\.The mechanism is internally coherent in a way that is worth stating, because it explains why the decay was necessary\. In the four hand\-inspected examples the licensor\-to\-function\-word distance is one or two tokens \(“ratherthan”\+1\+1; “turning theminto”\+2\+2; “neither quicknor”\+2\+2\), and the learned existential half\-life is∼\\sim1\.51\.5tokens\.*The decay tuned itself to the grammatical\-licensing window*: long enough to carry the membership from the licensor to the licensed position, short enough not to latch and pollute later predictions\. The three measurements of this section—recovers the licensing deficit \(§[9\.2](https://arxiv.org/html/2606.31845#S9.SS2)\), holds and deepens its structure \(§[9\.3](https://arxiv.org/html/2606.31845#S9.SS3)\), and forgets on a∼\\sim1\.51\.5\-token timescale \(§[9\.4](https://arxiv.org/html/2606.31845#S9.SS4)\)—thus compose into one legible operator:*fire on the licensor, decay for∼\\simone token, write the licensed word*\(Figure[5](https://arxiv.org/html/2606.31845#S9.F5)\)\. Each detector is, in effect, a learned, readable fragment of a transformer’s grammatical competence—of the kind the BLiMP suite tests directly\.

![Refer to caption](https://arxiv.org/html/2606.31845v1/figure_folog_comparison.png)\(a\)Comparison detector: fires on comparatives \(‘ rather’/‘ more’\) and predicts ‘ than’\.
![Refer to caption](https://arxiv.org/html/2606.31845v1/figure_folog_agentive.png)\(b\)Agentive detector: fires on passive participles \(‘ made’/‘ caused’\) and predicts ‘ by’\.
![Refer to caption](https://arxiv.org/html/2606.31845v1/figure_folog_directional.png)\(c\)Directional detector: fires on motion verbs \(‘ turning’/‘ put’\) and predicts ‘ into’\.
![Refer to caption](https://arxiv.org/html/2606.31845v1/figure_folog_disjunction.png)\(d\)Disjunction detector: fires on ‘ neither’/‘ either’ and predicts ‘ or’/‘ nor’\.

Figure 5:Four grammatical*licensing detectors*\(layer 9\), each a predictive operator\. Membership fires crisply on the grammatical*licensor*\(left spike\), the∼\\sim1\.51\.5\-token existential decay carries the signal forward over the licensing window, and the unit writes the*licensed*function word it predicts\. None fires on the function word it names\.
### 9\.6An honest ceiling: legibility is localized, not network\-wide

We resist over\-reading the result\. Quantifying*how much of the whole network*is human\-nameable— combining, per layer, the token\-explainability of the FFN units \(Boolean and membership, value\-vector to vocabulary against a matched\-random floor\) with the discernibility of the attention heads \(this model uses standard attention; we score each head as crisp\-and\-nameable or not\)—puts the figure at≈2%\\approx\\\!2\\%of all parameters, or≈4%\\approx\\\!4\\%if one excludes the deliberateGELU\\operatorname\{GELU\}gradient\-highway that carries no legibility claim\. This is a logit\-lens/single\-relation concentration proxy, not a causal autointerp score, and it should be read as a floor, not a verdict\. The honest summary is that legibility is*localized*: FFN nameability is monotonically back\-loaded, near the floor through L1–L7 and rising sharply to L11 \(25\.6%25\.6\\%; membership alone62\.5%62\.5\\%\), exactly where the licensing detectors live; attention nameability is the mirror image, present only as shallow positional heads \(L2–L4, L7\) with the deep half opaque\. So the claim of this section is precise and bounded: not an understandable network, but a*localized, interpretable\-by\-construction grammatical mechanism*—a bank of self\-forgetting quantifier units that, at the semantic layers, compute readable licensing—living inside an otherwise ordinary transformer, at no parameter cost and no loss of language\-model quality\. That is still a sharp advance over a standard FFN, whose interpretable\-by\-construction content is zero\.

## 10Discussion

#### What the results say together\.

The thread connecting capability, language modeling, and legibility is a single claim: a more*structured*FFN reasons more compactly and exposes*how*it computes, at a modest and well\-localized cost\. Capability makes the “compactly” concrete—a single bounded\-multiplicative layer is a more parameter\-efficient parity basis than strictly larger GELU \(Section[5](https://arxiv.org/html/2606.31845#S5)\)\. Language modeling shows the cost is real but small and, tellingly, that the capability is*dormant*for next\-token prediction, which does not reward products \(Section[6](https://arxiv.org/html/2606.31845#S6)\)\. Legibility delivers the “how”—explicit per\-unit logical form, a directly readable subset, causally\-used features—and the dynamics tie it back to capability: the logic*is*the reasoning, crystallizing when the task rewards it and relaxing when it does not \(Section[7\.6](https://arxiv.org/html/2606.31845#S7.SS6)\)\.

#### The sequence operator turns the limitations into the headline\.

Two of those findings are limitations, and they share a cause: the set operators are*within\-token*, so they cannot express that a feature occurred earlier in the sequence \(Section[9](https://arxiv.org/html/2606.31845#S9)\)\. Supplying the missing primitive—a fuzzy quantifier over the sequence with a*learned*forgetting rate—is the paper’s strongest result, and it is striking for three reasons\. First, the mechanism we add to*recover*the licensing deficit is the same one that, read out,*is*the licensing: the block whose job is sequence quantification both closes the grammar gap and exposes a bank of grammatical licensing detectors\. Second, the legibility no longer erodes—it holds and migrates into depth—so the dynamic relaxation we documented for the within\-token operators is specific to them, not to structured FFNs in general\. Third, and most telling for whether the operator belongs in a transformer at all, the decay*un\-learns*its own stickiness: from a sticky cumulative\-max initialization every unit learns a short,∼\\sim1\.51\.5\-token half\-life, and that half\-life turns out to match the one\-to\-two\-token distance from a grammatical licensor to the word it licenses\. The model was free to keep a permanent latch and instead learned a local, predictive operator—exactly the temporal logic a left\-to\-right language model can use—and the licensing detectors are what that operator looks like when it is read\.

#### What kind of logic this is—and what it is not\.

It is worth stating the scope of the “logic” precisely, because the architecture invites a natural misreading\. The set operators are*unary*predicates of a single token’s representation, and intersection and set\-difference are*propositional*connectives over them; the quantifiers \([9](https://arxiv.org/html/2606.31845#S9.E9)\)–\([10](https://arxiv.org/html/2606.31845#S9.E10)\) range over the*positions*of a unary predicate\. The fragment is therefore*monadic*predicate logic—unary predicates with quantification over the single sequence variable—*not*nn\-ary relational logic, and we make no claim to relational expressiveness\. This is a scope, not a shortfall, for two reasons\. First, monadic\-with\-sequence\-quantification is already strictly stronger than propositional: an existential over the prefix*binds a variable*—“a feature occurred at some earlier position”—which no within\-token propositional layer can express, and it is exactly the fragment a left\-to\-right model needs for the licensing dependencies we study\. Second, the dependency the quantifier captures—a licensor at one position to the word it licenses at another—is a relation between*positions*, bound by the decay\-quantifier, not a relation between argument slots at one token: the “second argument” is the other position, and the learned decay is what binds it\.NC\-FFN’s operators are*asymmetric*by construction \(A∖B≠B∖AA\\setminus B\\neq B\\setminus A; existential≠\\neqproportion\), which is also why a unit’s two operands do not degenerate into a redundant representation: measured directly, redundant operand pairs are under5%5\\%of units, the rest either keeping independent operands or cleanly retiring one to a single operand \(Table[5](https://arxiv.org/html/2606.31845#S7.T5)\)\. We therefore keep the FFN’s legible logic monadic and leave relational binding to attention; the contribution is the legibility and the discovered licensing structure, not expressiveness beyond the monadic fragment\. Explicitnn\-ary relational binding—role–filler structure rather than a single quantified variable—is a different mechanism again, the province of tensor\-product representations\(Smolensky,[1990](https://arxiv.org/html/2606.31845#bib.bib1)\)and the relational reading of tensor logic\(Domingos,[2025](https://arxiv.org/html/2606.31845#bib.bib2)\); how to incorporate such constructions effectively we leave to future work\.

#### Multiplicativity, bounding, negation: a decomposition of the design\.

The capability probe lets us assign roles to the propertiesNC\-FFNadds over a GELU FFN\.*Multiplicativity*is what does parity at all\.*Bounding*is what makes a multiplicative basis usable in width \(and so shallow\), at the price of depth\-composition—a genuine trade, not a strict improvement, since unbounded products are the better deep basis\.*Negation*\(the explicit complement\) adds a small shallow capability margin but is primarily a*legibility*feature: it turns the layer’s products into named set operations and is what makes the readable units read asand/and\-notpredicates\. We are deliberate that the headline capability is a property of the bounded multiplicative basis, shared with sigmoid\-bilinear; the negation’s distinctive contribution is interpretive\.

#### Limitations\.

The language\-model results are single\-seed at one scale \(125M\); the BLiMP deficit and the perplexity tie are robust, but the finer downstream numbers want replication\. The capability probe is a single synthetic family \(parity\) under a fixed optimization budget; it measures a real property—per\-parameter reasoning efficiency—but a longer budget would let unbounded products’ degree\-2L2^\{L\}advantage carry further at depth, and other reasoning families may apportion the width/depth axes differently\. And the directly\-legible units are a small minority; the bulk of operands are distributed, soNC\-FFNbuys an explicit combination*rule*for free but not, in general, monosemantic operands\. The quantifier results \(Section[9](https://arxiv.org/html/2606.31845#S9)\) are likewise single\-seed, though now reported across both epochs; the aggregate grammatical recovery, the learned\-forgetting statistics, and the licensing detectors are clear, but a seed sweep and a per\-subtask attribution of the BLiMP recovery remain to be run\. And the whole\-network legibility figure we report is a logit\-lens concentration proxy, not causal autointerp, and it is localized to a few late layers—a readable mechanism inside the network, not a readable network\.

#### A trainability boundary, and a path to a fully\-legible LM\.

The stabilizingGELU\\operatorname\{GELU\}partition is required, not merely convenient\. Section[8](https://arxiv.org/html/2606.31845#S8)characterizes a*Boolean\-fraction\-dependent*trainability horizon: at or below an even split the model trains a full epoch, but above it every run descends normally and then diverges abruptly—and the more Boolean the layer, the sooner, from∼\\sim136136k steps at75%75\\%down to∼\\sim1616k for a fully Boolean FFN\. Tellingly, the two obvious remedies—normalizing the residual write to unit magnitude, and adding a parallel linear highway—each only*delay*the divergence rather than prevent it, which points the cause at a saturated\-product gradient pathology \(the interaction of sigmoid saturation, composition, and the residual stream\) rather than the magnitude of the write\. This is the chief obstacle to an*end\-to\-end*\-legible language model, in which every FFN unit—not three\-quarters of the budget—is a named set operation\. Having characterized the instability, we regard*removing*it—most plausibly through a remedy aimed at the saturated\-product gradient rather than the residual write—as the most promising next step: it is the difference between a legible*component*and a legible*model*\.

#### Where the capability should pay off\.

Because the capability is dormant for next\-token prediction, the natural place to look for a*performance*win is tasks that reward multiplicative composition—symbolic, arithmetic, and algorithmic reasoning—rather than open\-web language modeling\. The width/depth dissociation also suggests a concrete design lever: bounded blocks where shallow, width\-cheap logic is wanted; unbounded \(or hybrid\) blocks where deep compositional reach is\. We leave both to future work\.

## 11Conclusion

We studied a parameter\-neutral transformer FFN whose hidden units are explicit fuzzy set operations, with a bounded complement that gives negation a clean positive form\. The choice is not free—it costs a little grammar and needs a stabilizing partition—but it buys two things an opaque activation does not\. It reasons more compactly: on a controlled probe, bounded multiplicative units are the most parameter\-efficient reasoning basis at shallow depth, and the properties we toggle \(multiplicativity, bounding\) cleanly govern the width and depth axes of that efficiency\. And it is legible: every unit is a named operation, a subset reads as predicates, and the network’s logical content is a dynamic readout of what its objective rewards, crystallizing at the moment of reasoning and eroding where only prediction is\. And when those within\-token operators are given an explicit*sequence*companion—a fuzzy quantifier free to learn how long to remember—the FFN’s hardest grammatical failures become its most legible mechanism: a bank of*self\-forgetting*quantifier units that, at the semantic layers, compute readable grammatical*licensing*, recover the grammar deficit at no parameter or quality cost, and each learn a memory just long enough to carry a licensor to the word it licenses\. That legibility is for now free only up to a partition: a stabilizingGELU\\operatorname\{GELU\}fraction is required, and training a fully Boolean, end\-to\-end\-legible model—removing the trainability boundary of Section[8](https://arxiv.org/html/2606.31845#S8)—is the open problem we leave\. A structured component, in short, lets us ask not just what a feed\-forward layer represents but how it combines—and, with an explicit sequence operator, how it*licenses*—and, increasingly, get an answer\.

## References

- Logic tensor networks\.Artificial Intelligence303\.Cited by:[§2\.2](https://arxiv.org/html/2606.31845#S2.SS2.p1.3),[§2\.3](https://arxiv.org/html/2606.31845#S2.SS3.p1.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn,et al\.\(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Cited by:[§1](https://arxiv.org/html/2606.31845#S1.p1.1),[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1)\.
- X\. Chen, Z\. Hu, and Y\. Sun \(2022\)Fuzzy logic based logical query answering on knowledge graphs\.InAAAI Conference on Artificial Intelligence,Cited by:[§2\.4](https://arxiv.org/html/2606.31845#S2.SS4.p1.3)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.3](https://arxiv.org/html/2606.31845#S4.SS3.p1.1)\.
- H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey \(2024\)Sparse autoencoders find highly interpretable features in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.31845#S1.p1.1),[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1)\.
- Y\. N\. Dauphin, A\. Fan, M\. Auli, and D\. Grangier \(2017\)Language modeling with gated convolutional networks\.International Conference on Machine Learning \(ICML\)\.Cited by:[§2\.1](https://arxiv.org/html/2606.31845#S2.SS1.p1.2)\.
- P\. Domingos \(2025\)Tensor logic: the language of AI\.arXiv preprint arXiv:2510\.12269\.Cited by:[§10](https://arxiv.org/html/2606.31845#S10.SS0.SSS0.Px3.p1.5)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Nanda, T\. Henighan,et al\.\(2022a\)Softmax linear units\.Transformer Circuits Thread\.Cited by:[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan,et al\.\(2022b\)Toy models of superposition\.Transformer Circuits Thread\.Cited by:[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1),[§7\.4](https://arxiv.org/html/2606.31845#S7.SS4.p1.9)\.
- L\. Gao, J\. Tow, S\. Biderman, S\. Black,et al\.\(2021\)A framework for few\-shot language model evaluation\.Note:EleutherAI lm\-evaluation\-harness, Zenodo,[https://doi\.org/10\.5281/zenodo\.5371628](https://doi.org/10.5281/zenodo.5371628)Cited by:[§4\.3](https://arxiv.org/html/2606.31845#S4.SS3.p1.1)\.
- F\. A\. Gers, J\. Schmidhuber, and F\. Cummins \(2000\)Learning to forget: continual prediction with LSTM\.Neural Computation12\(10\),pp\. 2451–2471\.Cited by:[§2\.3](https://arxiv.org/html/2606.31845#S2.SS3.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InEmpirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2606.31845#S1.p1.1),[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1)\.
- Y\. Huang and H\. Raza \(2025\)Semantic fusion with fuzzy\-membership features for controllable language modelling\.arXiv preprint arXiv:2509\.13357\.Cited by:[§2\.6](https://arxiv.org/html/2606.31845#S2.SS6.p1.1),[§2\.7](https://arxiv.org/html/2606.31845#S2.SS7.p1.1)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, Q\. N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández \(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InAssociation for Computational Linguistics \(ACL\),Cited by:[§4\.3](https://arxiv.org/html/2606.31845#S4.SS3.p1.1)\.
- M\. T\. Pearce, T\. Dooms, A\. Rigg, J\. Oramas, and L\. Sharkey \(2025\)Bilinear MLPs enable weight\-based mechanistic interpretability\.International Conference on Learning Representations \(ICLR\)\.Note:arXiv:2410\.08417Cited by:[§2\.1](https://arxiv.org/html/2606.31845#S2.SS1.p2.2),[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1),[§2\.7](https://arxiv.org/html/2606.31845#S2.SS7.p1.1),[§4\.5](https://arxiv.org/html/2606.31845#S4.SS5.p1.8)\.
- M\. T\. Pearce, T\. Dooms, and A\. Rigg \(2024\)Weight\-based decomposition: a case for bilinear MLPs\.arXiv preprint arXiv:2406\.03947\.Cited by:[§2\.1](https://arxiv.org/html/2606.31845#S2.SS1.p2.2)\.
- F\. Petersen, C\. Borgelt, H\. Kuehne, and O\. Deussen \(2022\)Deep differentiable logic gate networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.2](https://arxiv.org/html/2606.31845#S2.SS2.p1.3)\.
- F\. Petersen, H\. Kuehne, C\. Borgelt, J\. Welzel, and S\. Ermon \(2024\)Convolutional differentiable logic gate networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.2](https://arxiv.org/html/2606.31845#S2.SS2.p1.3),[§2\.7](https://arxiv.org/html/2606.31845#S2.SS7.p1.1)\.
- H\. Ren, W\. Hu, and J\. Leskovec \(2020\)Query2box: reasoning over knowledge graphs in vector space using box embeddings\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.4](https://arxiv.org/html/2606.31845#S2.SS4.p1.3)\.
- H\. Ren and J\. Leskovec \(2020\)Beta embeddings for multi\-hop logical reasoning in knowledge graphs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.31845#S1.p2.6),[§2\.4](https://arxiv.org/html/2606.31845#S2.SS4.p1.3),[§2\.7](https://arxiv.org/html/2606.31845#S2.SS7.p1.1)\.
- R\. Riegel, A\. Gray, F\. Luus, N\. Khan, N\. Makondo, I\. Y\. Akhalwaya,et al\.\(2020\)Logical neural networks\.arXiv preprint arXiv:2006\.13155\.Cited by:[§2\.2](https://arxiv.org/html/2606.31845#S2.SS2.p1.3),[§2\.3](https://arxiv.org/html/2606.31845#S2.SS3.p1.1)\.
- K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi \(2020\)WinoGrande: an adversarial winograd schema challenge at scale\.AAAI Conference on Artificial Intelligence\.Cited by:[§4\.3](https://arxiv.org/html/2606.31845#S4.SS3.p1.1)\.
- N\. Shazeer \(2020\)GLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§2\.1](https://arxiv.org/html/2606.31845#S2.SS1.p1.2),[§2\.7](https://arxiv.org/html/2606.31845#S2.SS7.p1.1),[§4\.5](https://arxiv.org/html/2606.31845#S4.SS5.p1.8)\.
- P\. Smolensky \(1990\)Tensor product variable binding and the representation of symbolic structures in connectionist systems\.Artificial Intelligence46\(1–2\),pp\. 159–216\.Cited by:[§10](https://arxiv.org/html/2606.31845#S10.SS0.SSS0.Px3.p1.5)\.
- A\. Tamkin, M\. Taufeeque, and N\. D\. Goodman \(2023\)Codebook features: sparse and discrete interpretability for neural networks\.arXiv preprint arXiv:2310\.17230\.Cited by:[§2\.5](https://arxiv.org/html/2606.31845#S2.SS5.p1.1)\.
- A\. Trask, F\. Hill, S\. Reed, J\. Rae, C\. Dyer, and P\. Blunsom \(2018\)Neural arithmetic logic units\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§2\.1](https://arxiv.org/html/2606.31845#S2.SS1.p3.1)\.
- E\. van Krieken, E\. Acar, and F\. van Harmelen \(2022\)Analyzing differentiable fuzzy logic operators\.Artificial Intelligence302\.Note:arXiv:2002\.06100Cited by:[§2\.2](https://arxiv.org/html/2606.31845#S2.SS2.p1.3),[§2\.3](https://arxiv.org/html/2606.31845#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2606.31845#S3.SS4.p1.2)\.
- A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. Bowman \(2020\)BLiMP: the benchmark of linguistic minimal pairs for english\.Transactions of the Association for Computational Linguistics \(TACL\)\.Cited by:[§2\.3](https://arxiv.org/html/2606.31845#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2606.31845#S4.SS3.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InAssociation for Computational Linguistics \(ACL\),Cited by:[§4\.3](https://arxiv.org/html/2606.31845#S4.SS3.p1.1)\.
- Z\. Zhang, J\. Wang, J\. Chen, S\. Ji, and F\. Wu \(2021\)ConE: cone embeddings for multi\-hop reasoning over knowledge graphs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.4](https://arxiv.org/html/2606.31845#S2.SS4.p1.3)\.