Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities

# Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities
Source: [https://arxiv.org/html/2608.12374](https://arxiv.org/html/2608.12374)
11institutetext:Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France22institutetext:Data ScienceTech Institute, Paris, France
22email:\{hanna\.abi\-akl,fabien\.gandon,pierre\.monnin\}@inria\.fr, faron@i3s\.unice\.fr###### Abstract

Language models \(LMs\) struggle with logical tasks like reasoning on syllogisms\. It has been shown that Knowledge Representation \(KR\) plays a crucial role in expressing input information to help models solve tasks\. This observation motivates our study of the impact of different formal KR notations on syllogistic reasoning by extending the FOLIO and P\-FOLIO datasets\. Our experiments on Small Language Models \(SLMs\) in Supervised Fine\-Tuning \(SFT\) and Zero\-Shot \(ZS\) settings show that the choice of input notation can yield performances competitive with natural language while enabling faster inference\. We also propose a syllogistic categorization method \(SEF\) and use it to enrich ZS prompts with logical definitions, which boost reasoning in small models\. We open\-source our framework, Common Logic Grammar Construction \(CLGC\), as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories\.

## 1Introduction

Advances in Artificial Intelligence \(AI\) techniques have seen Language Models \(LMs\) hit new heights in solving tasks of increasing complexity\. However, logic tasks like planning or puzzle solving still represent a challenging frontier\. In this paper, we focus on generalized syllogistic reasoning covering multiple forms of syllogisms with new datasets and LM\-based techniques to increase model robustness and consistency\[[3](https://arxiv.org/html/2608.12374#bib.bib3),[6](https://arxiv.org/html/2608.12374#bib.bib6),[7](https://arxiv.org/html/2608.12374#bib.bib7),[17](https://arxiv.org/html/2608.12374#bib.bib17)\]\. Past research has shown that the complexity of this task starts with the dataset, where current datasets suffer from shortcomings like insufficient human involvement, size, diversity, or generalization\[[9](https://arxiv.org/html/2608.12374#bib.bib9),[10](https://arxiv.org/html/2608.12374#bib.bib10),[20](https://arxiv.org/html/2608.12374#bib.bib20),[27](https://arxiv.org/html/2608.12374#bib.bib27),[13](https://arxiv.org/html/2608.12374#bib.bib13),[5](https://arxiv.org/html/2608.12374#bib.bib5)\]\. Moreover, recent neuro\-symbolic approaches have used symbolic data representations like First\-Order Logic \(FOL\) due to their interpretable and unambiguous nature as opposed to Natural Language \(NL\)\. These approaches have shown promising results in boosting model performance on reasoning tasks, most notably small and medium LMs\[[18](https://arxiv.org/html/2608.12374#bib.bib18),[31](https://arxiv.org/html/2608.12374#bib.bib31)\]\. However, to the best of our knowledge, syllogistic reasoning is currently evaluated only in NL and FOL notations, whereas other shades of Knowledge Representation \(KR\) languages with varying levels of abstraction between natural and symbolic exist to represent logical data\. Additional metadata about the reasoning task, e\.g\. categories of syllogisms, could also be considered to support LM reasoning\. Varying the input data representation and studying its effects on logical reasoning thus appears as a necessary challenge to evaluate the limits of LMs in scenarios requiring complex abstract thinking\[[30](https://arxiv.org/html/2608.12374#bib.bib30)\]\. These observations motivate our work, where we aim to address the following Research Questions \(RQs\):

RQ1\.How do different input formal notations impact LM syllogistic reasoning?

RQ2\.How does grouping syllogisms in pre\-defined categories and prompting LMs with this information affect their reasoning abilities?

We propose to answer these RQs by providing a protocol comprising automatic formal notation generation, syllogism category definition and classification, and evaluation of Small Language Models \(SLMs\) on syllogism datasets in Supervised Fine\-Tuning \(SFT\) and Zero\-Shot \(ZS\) settings\. We specifically focus on SLMs due to their frugality and performances on similar tasks\[[1](https://arxiv.org/html/2608.12374#bib.bib2),[18](https://arxiv.org/html/2608.12374#bib.bib18),[31](https://arxiv.org/html/2608.12374#bib.bib31)\]\. Our contribution to the field is threefold:

- •We introduce the Common Logic Grammar Construction framework for generating syllogisms in different formal notations as well as categorizing them based on their structure; we open\-source it as a Python package\.
- •We release an enriched, extended version of the FOLIO and P\-FOLIO reasoning datasets with formal notations for our experiments\.
- •We evaluate the impact of several formal notations on the performance of LM on syllogistic reasoning and establish different trends based on the abstraction level of the notation\.

The rest of this paper is organized as follows: we discuss the related work in Section[2](https://arxiv.org/html/2608.12374#S2), present our methodology in Section[3](https://arxiv.org/html/2608.12374#S3), describe our experimental setup in Section[4](https://arxiv.org/html/2608.12374#S4), discuss our findings in Section[5](https://arxiv.org/html/2608.12374#S5)and conclude in Section[6](https://arxiv.org/html/2608.12374#S6)\.

## 2Related Work

### 2\.1Language Models and Reasoning

Logic in AI deals with tasks involving abstraction like syllogistic reasoning\[[29](https://arxiv.org/html/2608.12374#bib.bib29)\]\. Current neuro\-symbolic approaches have used symbolic structures to improve language model reasoning\. Advanced prompting techniques with rule templates have shown increases in performance and more control on reasoning bias in syllogisms\[[21](https://arxiv.org/html/2608.12374#bib.bib21),[28](https://arxiv.org/html/2608.12374#bib.bib28)\]\. Other methods use Chain\-Of\-Thought \(COT\) to generate reasoning steps and explanations as a self\-correction mechanism for LMs\[[32](https://arxiv.org/html/2608.12374#bib.bib32),[14](https://arxiv.org/html/2608.12374#bib.bib14)\]\. Recently, hybrid architectures have used NL syllogisms and FOL translations with a theorem prover to guide model reasoning\[[19](https://arxiv.org/html/2608.12374#bib.bib19),[12](https://arxiv.org/html/2608.12374#bib.bib12),[16](https://arxiv.org/html/2608.12374#bib.bib15),[11](https://arxiv.org/html/2608.12374#bib.bib11)\]\. Our work leverages different input representations as logical basis to improve reasoning\. Training SLMs with different input representations has shown promising results compared to NL baselines on syllogisms\[[2](https://arxiv.org/html/2608.12374#bib.bib1),[1](https://arxiv.org/html/2608.12374#bib.bib2)\]\. In that perspective, we use existing yet under\-explored KR notation families proposed in the literature, namely: the Common Logic Interchange Format \(CLIF\) and Conceptual Graph Interchange Format \(CGIF\)\[[26](https://arxiv.org/html/2608.12374#bib.bib23),[25](https://arxiv.org/html/2608.12374#bib.bib24),[24](https://arxiv.org/html/2608.12374#bib.bib25),[23](https://arxiv.org/html/2608.12374#bib.bib26)\]from the Common Logic \(CL\) suite, Tensor Functor Logic Plus \(TFLPLUS\)\[[22](https://arxiv.org/html/2608.12374#bib.bib22),[4](https://arxiv.org/html/2608.12374#bib.bib4),[15](https://arxiv.org/html/2608.12374#bib.bib16)\]from Plus\-Minus Algebra, and CLINGO\[[8](https://arxiv.org/html/2608.12374#bib.bib8)\]from Answer Set Programming \(ASP\) languages\. To the best of our knowledge, no other method in the literature adopts and evaluates the impact of KR notations on syllogistic reasoning\.

### 2\.2LM Experimentation on Syllogisms

The nature of LM experimentation and the need for an efficient iterative approach due to the sensitivity of models to different parameters such as input data justifies the need for standardized frameworks for reasoning tasks\. To the best of our knowledge, no such framework exists today to study syllogisms data in different representations\. The closest initiatives are solvers in Python111[https://github\.com/czrptr/syllogism\-solver](https://github.com/czrptr/syllogism-solver),[https://github\.com/mhtess/syllogism](https://github.com/mhtess/syllogism)and Lua222[https://pypi\.org/project/syllogistic/](https://pypi.org/project/syllogistic/)but they operate with syllogisms in NL\. To address this gap, we propose the Common Logic Grammar Construction framework \(CLGC\)333[https://pypi\.org/project/clgc/](https://pypi.org/project/clgc/), a Python package to automatically manipulate syllogisms in different formal notations\.

## 3Syllogistic Reasoning: From Formal Notation Generation to LM Evaluation

### 3\.1CLGC Formal Notation Generation

#### 3\.1\.1Notation Selection Criteria\.

We define the following criteria:

Verbosity\.Our set should include compact \(i\.e\. with minimal syntactic variations\) and verbose \(i\.e\. with many syntactic variations\) notations\. Greater verbosity leaves more room for ambiguity when expressing a syllogism\.

Frequency\.Frequency is relative to the LMs used in our experimental setup \(Section[4](https://arxiv.org/html/2608.12374#S4)\)\. Since LM pre\-training is not always fully disclosed, we select notations that are frequently seen by LMs \(e\.g\. NL\) versus notations we assume are much less seen \(e\.g\. CLIF\), to the best of our estimation\.

Abstractness\.Abstractness defines the syntactic complexity of a notation, ranging from natural \(i\.e\. NL\) to symbolic, mathematical notations\.

Finiteness\.A hard requirement is for notations to have a finite vocabulary and definable grammar\.

#### 3\.1\.2Generation Process\.

To generate a formal notation, we start from syllogisms in FOL\. FOL meets the above defined criteria and has a defined grammar in Backus\-Naur Form \(BNF\)\. We reuse the algorithm previously introduced in\[[2](https://arxiv.org/html/2608.12374#bib.bib1)\]to automatically generate a formal notation from FOL\. The algorithm consists in first generating the Abstract Syntax Tree \(AST\) from the FOL BNF grammar and then constructing an equivalent tree in the target notation using the notation’s BNF grammar that we manually implement in accordance with the above defined finiteness criteria\. From the resulting AST, a parser reconstructs statements in the target notation while applying syntactic rules specific to the notation \(e\.g\. spacing, parentheses\)\. To make the process reproducible, we packaged and open\-sourced our pipeline in the Python CLGC444Named as an ode to John Sowalibrary555[https://github\.com/HannaAbiAkl/clgc](https://github.com/HannaAbiAkl/clgc)\.

#### 3\.1\.3Generated Notations\.

Figure[1](https://arxiv.org/html/2608.12374#S3.F1)shows the current formal notations supported in CLGC\. The KR suite contains families of formal notations like CL languages \(i\.e\. CLIF, CGIF\), abstract notations like the Plus\-Minus Algebra family \(i\.e\. TFLPLUS\), and the ASP family \(i\.e\. CLINGO\)\.

![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig19.png)Figure 1:CLGC Notations\.The Miniature FOL \(MINIFOLx\) suite is a family of previously unseen lightweight variants of FOL that we introduce as a baseline for studying how minor syntactic variations in notation affect LM syllogistic reasoning performance, thereby enabling to assess sensitivity to the Frequency criterion\. Precisely, MINIFOL replaces some FOL vocabulary \(i\.e\. symbols\) with vocabulary that is more common for LMs \(i\.e\. words\)\. Here,xdenotes a specific variant \(e\.g\., MINIFOL1\)\. We extend the definition of MINIFOL from\[[1](https://arxiv.org/html/2608.12374#bib.bib2)\]to the variants introduced in this work:

MINIFOL\.Replaces FOL operators \(namely:∀\\forall,⊕\\oplus,→\\rightarrow,¬\\neg,∃\\exists,∧\\wedge,∨\\vee\) by strings \(respectively: all, ˆ , :\-, ˜ , some, &, \|\)\.

MINIFOL2\.Eliminates the∃\\existsquantifier in MINIFOL\.

MINIFOL3\.Replaces the¬\\negoperator withnotin MINIFOL\.

MINIFOL4\.Replaces the∧\\wedgeoperator with,in MINIFOL\.

### 3\.2Syllogism Evaluation Framework \(SEF\)

SEF is a categorization mechanism previously introduced in\[[2](https://arxiv.org/html/2608.12374#bib.bib1)\]to classify syllogisms based on their structure in one of the four following categories:

Hypothetical\.Syllogism containing an implication

Disjunctive\.Syllogism containing a disjunction

Categorical\.Syllogism of exactly 2 premises not belonging to the aforementioned categories

Complex\.Syllogism not belonging to any other category

Each category is associated with a definition leveraged by CLGC to automatically detect a syllogism’s SEF category\. These definitions can also be used to guide syllogistic reasoning\. Table[1](https://arxiv.org/html/2608.12374#S3.T1)illustrates the SEF categories\.

Table 1:SEF category examples from P\-FOLIO\-KR\.SEF CategorySyllogism exampleComplexRenamedAs\(fortCarillon, fortTiconderoga\)Built\(pierredeRigauddeVaudreuil, fortCarillon\)LocatedIn\(fortCarillon, newFrance\)¬\\negLocatedIn\(newFrance, europe\)CategoricalSportingEvent\(olympics\)LastSummerOlympics\(tokyo\)MostMedals\(unitedStates, tokyo\)Hypothetical∀\\forallx \(Square\(x\)→\\rightarrowFourSided\(x\)\)∀\\forallx \(FourSided\(x\)→\\rightarrowShape\(x\)\)∀\\forallx \(Square\(x\)→\\rightarrowShape\(x\)\)DisjunctiveFlyTo\(susan, lgaAirport\)∀\\forallx∀\\forally \(FlyFrom\(x, y\)⊕\\oplusFlyTo\(x, y\)\)FlyFrom\(john, lgaAirport\)
### 3\.3Evaluating LM Reasoning Capability

We formalize the task as follows\. A syllogismSis a logical problem defined by a set of statements called premises and a conclusion\. The goal is to verify if the conclusion logically follows from the premises by labeling the conclusionTrue\(i\.e\. valid\),False\(i\.e\. invalid\), orUnknown\(i\.e\. inconclusive\)\.

We conduct our evaluation in SFT and ZS settings\. In SFT, we aim to evaluate how well a model can learn a formal notation by training it on syllogisms in that notation and their truth labels\. The model is then tasked to predict the truth label of syllogisms formalized in the same notation\. In ZS, we consider two scenarios\.Scenario 1prompts the model to predict the truth label of a syllogism in a formal notation given the syllogism and the BNF grammar of the notation\.Scenario 2adds the SEF category of the syllogism with its definition, and an example to the prompt\. The aim of providing this additional information is to facilitate the understanding of logical problems and guide the reasoning process\. Below is the template of each prompt scenario for the FOL notation\.

Scenario 1You are an expert logician\. You are given a syllogism in FOL with premises between <PREMISES\></PREMISES\> and conclusion between <CONCLUSION\></CONCLUSION\> tags\. The FOL BNF grammar to understand and reason in the language is given in the <GRAMMAR\></GRAMMAR\> tags\. <GRAMMAR\>…</GRAMMAR\> <PREMISES\>…</PREMISES\> <CONCLUSION\>…</CONCLUSION\> Classify the conclusion as “T” if true, “F” if false or “U” if unknown based on the premises\. Present your answer only between <output\></output\> tags\.

Scenario 2You are an expert logician\. You are given a syllogism in FOL with premises between <PREMISES\></PREMISES\> and conclusion between <CONCLUSION\></CONCLUSION\> tags\. The FOL BNF grammar to understand and reason in the language is given in the <GRAMMAR\></GRAMMAR\> tags\. <GRAMMAR\>…</GRAMMAR\> <PREMISES\>…</PREMISES\> <CONCLUSION\>…</CONCLUSION\> You are also given the category of the syllogism to help you understand it: Disjunctive\. A disjunctive syllogism contains “∨\\vee” or “⊕\\oplus”\. Here is an example: <PREMISES\> …</PREMISES\> <CONCLUSION\> …</CONCLUSION\> Classify the conclusion as “T” if true, “F” if false or “U” if unknown based on the premises\. Present your answer only between <output\></output\> tags\.

## 4Experimental Setup

Our experiments extend previous work in\[[2](https://arxiv.org/html/2608.12374#bib.bib1),[1](https://arxiv.org/html/2608.12374#bib.bib2)\]in terms of datasets and setup\.

### 4\.1Datasets

To evaluate LM syllogistic reasoning, we consider the FOLIO\[[9](https://arxiv.org/html/2608.12374#bib.bib9)\]and P\-FOLIO\[[10](https://arxiv.org/html/2608.12374#bib.bib10)\]human\-curated syllogism datasets in NL and FOL\. We use CLGC to extend them and create publicly available KR versions respectively denoted FOLIO\-KR666[https://huggingface\.co/datasets/HannaAbiAkl/FOLIO\-KR](https://huggingface.co/datasets/HannaAbiAkl/FOLIO-KR)and P\-FOLIO\-KR777[https://huggingface\.co/datasets/HannaAbiAkl/P\-FOLIO\-KR](https://huggingface.co/datasets/HannaAbiAkl/P-FOLIO-KR)\. These extended versions contain the syllogisms generated from FOL into CLIF, CGIF, CLINGO, TFLPLUS, and our custom MINIFOLxnotations\. Table[2](https://arxiv.org/html/2608.12374#S4.T2)shows an example of a syllogism from P\-FOLIO\-KR\. We also use CLGC to generate the SEF categories for each syllogism in FOLIO and P\-FOLIO and include them in our KR extended versions\. Table[3](https://arxiv.org/html/2608.12374#S4.T3)summarizes label and SEF category statistics for both datasets\.

Table 2:CLGC notation example from P\-FOLIO\-KR\.NotationExampleFOL∀\\forallx \(Has\(x, flu\)→\\rightarrowHas\(x, influenza\)\)CLIFforall x \(has\(x, flu\) implies has\(x, influenza\)\)CLINGOforall \(has\(x, flu\) \-: has\(x, influenza\)\)CGIF@every \*x \[\(has\[\(?x flu\)\] has\[\(?x influenza\)\]\)\]MINIFOL2all:x \(has\(x, flu\) :\- has\(x, influenza\)\)TFLPLUS\-\(\+H0\-\+H0\)Table 3:Statistics of the extended FOLIO\-KR and P\-FOLIO\-KR datasets\.DatasetTotalLabelSEFTrueFalseUnknownCategoricalHypotheticalDisjunctiveComplexFOLIO\-KR12044603513931767653467P\-FOLIO\-KR3011226911091340239

### 4\.2Protocol

We treat the syllogistic reasoning problem as a multi\-class classification task with the values \{True, False, Unknown\}\. Experiments were performed on SLMs in SFT and ZS settings on P\-FOLIO\-KR and FOLIO\-KR\. SFT experiments were performed on A100 GPUs and ZS experiments on L4 GPUs\.

#### 4\.2\.1Supervised Fine\-Tuning\.

In SFT, the dataset is split into the Train, Val and Test sets\. In the Train and Val sets, each syllogism \(i\.e\. premises and conclusion\) in a notation is associated to its truth label\. In the Test set, the truth labels are omitted and the model predicts them for each syllogism during inference\. The FOLIO\-KR and P\-FOLIO\-KR datasets are split into stratified frozen Train, Val, and Test sets for reproducibility\. Stratification balances the splits and ensures all sets have similar truth label and SEF category distributions\. Table[4](https://arxiv.org/html/2608.12374#S4.T4)shows the statistics for the labels and SEF categories in the frozen splits\.

We build on previous SLM results\[[9](https://arxiv.org/html/2608.12374#bib.bib9),[10](https://arxiv.org/html/2608.12374#bib.bib10)\]by limiting our selection to Flan\-T5\-small888[https://huggingface\.co/google/flan\-t5\-small](https://huggingface.co/google/flan-t5-small)and Flan\-T5\-large999[https://huggingface\.co/google/flan\-t5\-large](https://huggingface.co/google/flan-t5-large)to study notation\-based reasoning on smaller, frugal language models as defined in\[[30](https://arxiv.org/html/2608.12374#bib.bib30)\]\. The Flan\-T5 encoder\-decoder architecture makes the model a good candidate for both understanding and generating text which is required for syllogistic reasoning\. Furthermore, Flan\-T5 has already exhibited good performances on this task\[[9](https://arxiv.org/html/2608.12374#bib.bib9),[10](https://arxiv.org/html/2608.12374#bib.bib10)\]with respect to other similar models in size and architecture \(e\.g\. RoBERTa\)\. Training is performed on 5 epochs with a batch size of 4\. The models are initialized with a seed to ensure reproducibility\. We list all training parameters on GitHub101010[https://github\.com/HannaAbiAkl/clgc/tree/main/experiments/folio/notebooks](https://github.com/HannaAbiAkl/clgc/tree/main/experiments/folio/notebooks)\.

Table 4:Frozen split statistics\.DatasetSplitTotalLabelSEFTrueFalseUnknownCategoricalHypotheticalDisjunctiveComplexP\-FOLIO\-KRTrain1445833533421116Val96392235661569Test6125142203454FOLIO\-KRTrain8003162252591143432314Val20172646531311570Test20372626931110683

#### 4\.2\.2Zero\-Shot\.

In Zero\-Shot, the model is given the syllogism and a prompt instructing it to use logical reasoning to predict the truth value from the possible labels\. For these experiments, we also restrict ourselves to small, decoder\-based models ideal for text generation \(i\.e\. less than 10 billion parameters\[[30](https://arxiv.org/html/2608.12374#bib.bib30)\]\)\. Our selection comprises Gemma\-2\-2b\-it111111[https://huggingface\.co/google/gemma\-2\-2b\-it](https://huggingface.co/google/gemma-2-2b-it), Llama\-3\.2\-3b\-instruct121212[https://huggingface\.co/meta\-llama/Llama\-3\.2\-3B\-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct), and Phi\-3\.5\-mini\-instruct131313[https://huggingface\.co/microsoft/Phi\-3\.5\-mini\-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)\. Each model has its particularity aside from being pre\-trained on different corpora: Gemma is an all\-around general model, Llama is open\-source, and Phi is specialized in reasoning and mathematical tasks\. All models are trained on Scenarios 1 and 2 with the prompts of Section[3\.3](https://arxiv.org/html/2608.12374#S3.SS3)\.

## 5Results and Discussion

Since our datasets are unbalanced in truth labels \(Table[3](https://arxiv.org/html/2608.12374#S4.T3)\), we measure and report model performance using the F1 score in our SFT and ZS experiments\. In ZS, we also compute the Absolute Gain \(AG\) to measure the change in F1 between Scenario 1 \(S1\) and 2 \(S2\) prompts:A​GF​1=F​1S2−F​1S1AG\_\{F1\}=F1\_\{\\text\{S2\}\}\-F1\_\{\\text\{S1\}\}\.

### 5\.1Supervised Fine\-Tuning

#### 5\.1\.1Notation performance depends on model and dataset\.

In Table[5](https://arxiv.org/html/2608.12374#S5.T5), we observe that for P\-FOLIO\-KR, Flan\-T5\-small performs best on abstract notations \(e\.g\. TFLPLUS\) and combinations thereof \(e\.g\. CLIF \+ TFLPLUS, NL \+ CLIF \+ TFLPLUS\)\. TFLPLUS performs almost 3 times better than NL with only limited SFT training \(144 and 96 syllogisms respectively in Train and Val sets\) compared to the model pre\-training on NL\. FOL and FOL\-like notations \(i\.e\. MINIFOL2, CLINGO\) perform midway between NL and TFLPLUS\. Of all notations, TFLPLUS has the simplest syntax, with\+\+and−\-tokens already seen by Flan\-T5\-small’s tokenizer during pre\-training\. We can hypothesize that this is the reason why the model learns the notation well as opposed to MINIFOL2 for example\. Flan\-T5\-large’s performance shows a different trend, where NL is the best notation followed by CLIF which is a more compact version of NL\. The model still performs better on TFLPLUS than on FOL\-like notations\. We can hypothesize that the additional NL data seen by Flan\-T5\-large in pre\-training compared to Flan\-T5\-small is the reason for the boost in performance in NL, and that some symbols in FOL and FOL\-like notations \(e\.g\. quantifiers\) are harder to learn, due to their under\-representation in the pre\-training dataset\.

For FOLIO\-KR, we observe that Flan\-T5\-small performs best on a combination of NL and CLIF\. FOL and FOL\-like notations \(CLINGO, CGIF\) rank midway in performance while TFLPLUS is among the worst performers\. FOLIO\-KR’s larger Train and Val splits \(800 and 201 syllogisms respectively\) compared to P\-FOLIO\-KR could cause this boost in performance of notations with well\-seen tokens like NL, CLIF and NL \+ CLIF\. This increase did not help notations like TFLPLUS and MINIFOL which rely heavily on potentially less seen symbolic tokens \(i\.e\. algebraic and boolean operators\)\. This might also explain the relative good performance ofMINIFOLx\\text\{MINIFOL\}\_\{x\}, which eliminate symbols compared with MINIFOL\. For Flan\-T5\-large, we observe results on FOLIO\-KR similar to those of Flan\-T5\-small\. NL is the best notation, followed by CLIF\. NL \+ CLIF ranks between the two notations\. MINIFOL2 and CLINGO, which share similar abstraction and complexity \(e\.g\. shared symbols like:\-\) rank close behind\. We can hypothesize that FOL and CGIF –the most complex notations \(i\.e\. containing the most symbols\)– and TFLPLUS –the most abstract one \(i\.e\. everything is represented with\+\+and−\-tokens\)– are the hardest to learn at scale\.

Table 5:Flan\-T5 SFT results\. Best results are in bold, second\-best are underlined\.ModelNotationF1SmallCLIF \+ TFLPLUS0\.367TFLPLUS0\.350NL \+ CLIF \+ TFLPLUS0\.339CLIF0\.298CGIF0\.295FOL0\.289CLINGO0\.276MINIFOL20\.239NL \+ CLIF0\.187NL0\.126LargeNL0\.688CLIF0\.510NL \+ CLIF \+ TFLPLUS0\.508NL \+ CLIF0\.490TFLPLUS0\.460CGIF0\.368MINIFOL20\.324FOL0\.278CLINGO0\.272

\(a\)P\-FOLIO\-KR
ModelNotationF1SmallNL \+ CLIF0\.447NL0\.435CLIF0\.410NL \+ FOL0\.395FOL0\.382MINIFOL20\.362MINIFOL30\.357MINIFOL40\.356CLINGO0\.353CGIF0\.339TFLPLUS0\.291MINIFOL0\.190LargeNL0\.658NL \+ CLIF0\.638CLIF0\.614CLINGO0\.601MINIFOL20\.581FOL0\.506TFLPLUS0\.534CGIF0\.181MINIFOL0\.177

\(b\)FOLIO\-KR

#### 5\.1\.2Performance increases with model size independently of abstraction\.

Figure[2](https://arxiv.org/html/2608.12374#S5.F2)shows the performance trend of different notations based on model size on P\-FOLIO\-KR and FOLIO\-KR in SFT\. On P\-FOLIO\-KR, we observe that all notations, except FOL, exhibit an upward trend when scaling Flan\-T5\. We also observe that Flan\-T5\-small performs better on more abstract notations but the trend changes when the model is scaled\. Similarly, on FOLIO\-KR, performance increases for all notations with scaling, with FOL experiencing a slight lesser growth\. We can hypothesize that the behavior of FOL is due to the complexity of its syntax in terms of tokens compared to more abstract but syntactically simpler notations \(i\.e\. TFLPLUS\)\. This may also explain FOL’s downward trend on P\-FOLIO\-KR since P\-FOLIO\-KR’s syllogisms are designed to be longer and more complex in reasoning depth than FOLIO\-KR’s\[[10](https://arxiv.org/html/2608.12374#bib.bib10)\]\.

![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig20.png)\(a\)P\-FOLIO\-KR\.
![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig6.png)\(b\)FOLIO\-KR\.

Figure 2:Model size impact on notation performance for Flan\-T5\.
#### 5\.1\.3Combining natural and symbolic notations yields more conservative reasoning\.

Figure[3](https://arxiv.org/html/2608.12374#S5.F3)compares the per\-label accuracy of the best notations \(i\.e\. NL, CLIF and NL \+ CLIF\) with Flan\-T5 small and large models on FOLIO\-KR\. For the label True, we observe that CLIF is the worst performer in the small and large models\. Results also show that while NL \+ CLIF performs better than NL in Flan\-T5\-small, it does not outperform it in the large model\. We hypothesize that CLIF makes the model reasoning more conservative by limiting True predictions\. This is further substantiated by the results on False and Unknown\. For the False label, CLIF benefits from the biggest increase in performance when scaling from small to large and outperforms NL \+ CLIF in Flan\-T5\-large\. One hypothesis is that the increase in performance on the False label is related to the compactness of CLIF compared to NL which makes it favor reasoning by refutation \(i\.e\. predicting False\)\. Results on the Unknown label show that NL \+ CLIF bests the other notations at small and large scales\. We observe that at small scale, CLIF performs better than NL\. We can hypothesize that combining a compact notation like CLIF with NL improves model reasoning in uncertainty\. This may be seen as the “hardest” form of reasoning since it cannot validate \(i\.e\. True\) or refute \(i\.e\. False\) based on the premises and conclusion at hand\. It can be argued that this type of reasoning requires more abstraction which NL \+ CLIF allows\. In view of the general tendency of LMs to hallucinate instead of stating they do not know the answer, the global conservative behavior of CLIF \(or NL \+ CLIF\) could be seen as a potential remediation, where predicting Unknown could also be seen as the best compromise\.

![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig7.png)\(a\)True label\.
![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig8.png)\(b\)False label\.
![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig9.png)\(c\)Unknown label\.

Figure 3:Flan\-T5\-small and large NL, CLIF and NL \+ CLIF per\-label accuracy on FOLIO\-KR SFT\.
#### 5\.1\.4NL \+ CLIF as a conservative SFT reasoning language representation\.

![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig10.png)\(a\)True label\.
![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig11.png)\(b\)False label\.
![Refer to caption](https://arxiv.org/html/2608.12374v1/figures/fig12.png)\(c\)Unknown label\.

Figure 4:Flan\-T5\-large NL, CLIF and NL \+ CLIF label prediction distribution on FOLIO\-KR SFT\.In Figure[4](https://arxiv.org/html/2608.12374#S5.F4), we observe that combining NL and CLIF reverses the trend of misclassification on the True label\. NL \+ CLIF favors False as a mistake contrary to NL and CLIF\. For the False label, we observe that NL \+ CLIF is more balanced in misclassifying False as either True or Unknown compared to NL, with CLIF alone being the most balanced notation\. Results also show that CLIF is the best at classifying False, which strengthens our claim that this compact notation improves a specific type of reasoning \(i\.e\. reasoning by refutation\) compared to NL\. For the Unknown label, we observe that NL and NL \+ CLIF are tied in performance but CLIF and NL \+ CLIF are more likely to misclassify the label as False than NL\. Since label distribution in FOLIO\-KR largely favors True \(Table[4](https://arxiv.org/html/2608.12374#S4.T4)\), we hypothesize that introducing a more compact notation \(i\.e\. CLIF\) and combinations \(i\.e\. NL \+ CLIF\) alters model reasoning\. Figure[3](https://arxiv.org/html/2608.12374#S5.F3)supports this hypothesis as scaling the model shows that NL is the best in predicting True, CLIF alone bests NL \+ CLIF on False, and NL \+ CLIF is consistently the best at predicting Unknown\. This is a promising avenue for combining KR notations to refine model reasoning, possibly in a multi\-stage neuro\-symbolic pipeline with NL as a first step and refinement in CLIF or NL \+ CLIF\.

### 5\.2Zero\-Shot

Table 6:ZS results for S1 and S2\. Best results are in bold, second\-best are underlined\.DatasetModelNotationF​1S1F1\_\{\\text\{S1\}\}F​1S2F1\_\{\\text\{S2\}\}AGP\-FOLIO\-KRGemma\-2\-2b\-itNL0\.3530\.464\+0\.111FOL0\.3020\.400\+0\.098CLIF0\.3020\.368\+0\.066CGIF0\.3730\.221\-0\.152CLINGO0\.4010\.343\-0\.058MINIFOL20\.2950\.364\+0\.069TFLPLUS0\.2980\.247\-0\.051NL \+ CLIF0\.4030\.375\-0\.048Llama\-3\.2\-3b\-instructMINIFOL20\.0410\.049\+0\.008FOL0\.0210\.048\+0\.027CLIF0\.0270\.020\-0\.007NL0\.0350\.038\+0\.003CGIF0\.0450\.043\-0\.002CLINGO0\.0170\.045\+0\.028TFLPLUS0\.0160\.014\-0\.002Phi\-3\.5\-mini\-instructCLIF0\.2060\.141\-0\.065NL0\.1010\.143\+0\.042FOL0\.1630\.123\-0\.040CGIF0\.0590\.066\+0\.007CLINGO0\.0770\.186\+0\.109MINIFOL20\.1290\.042\-0\.087TFLPLUS0\.0790\.045\-0\.034FOLIO\-KRGemma\-2\-2b\-itNL \+ CLIF0\.2340\.316\+0\.082NL0\.2240\.311\+0\.087CLIF0\.2660\.272\+0\.006CLINGO0\.2700\.265\-0\.005CGIF0\.1780\.208\+0\.030FOL0\.2100\.137\-0\.073MINIFOL20\.1930\.258\+0\.065TFLPLUS0\.1840\.1840

#### 5\.2\.1The choice of KR notation depends on model and training paradigm\.

Comparing Tables[5](https://arxiv.org/html/2608.12374#S5.T5)and[6](https://arxiv.org/html/2608.12374#S5.T6)shows variations in notation performance with model and training paradigm\. In SFT, NL, CLIF and the combination thereof \(i\.e\. NL \+ CLIF\) seem to be the best\-suited for models like Flan\-T5\. In ZS, there is no prior fine\-tuning phase on the notation before inference and model performance varies based on the notation abstraction and the model’s initial pre\-training data\. To the best of our knowledge, Phi is trained on synthetic and logical data \(e\.g\. mathematical\) which might make it more suited for reasoning in abstract notations \(e\.g\. CLINGO\) with SEF descriptions\. Gemma is, to the best of our knowledge, mostly pre\-trained on text data which fits the hypothesis that it should perform better with NL and NL\-like \(i\.e\. CLIF\) notations with SEF descriptions in those notations\. One possible explanation for Llama’s performance on FOL\-like notations may be its distilled pre\-training data which is highly filtered content from the web, making it less averse to notations with symbolic tokens than models like Gemma\. These observations strengthen our claim in Section[5\.1](https://arxiv.org/html/2608.12374#S5.SS1)regarding multi\-stage model design based on notation and learning paradigm\.

#### 5\.2\.2SEF description impact varies with model and notation\.

Table[6](https://arxiv.org/html/2608.12374#S5.T6)shows the ZS results for scenarios S1 and S2 on P\-FOLIO\-KR and FOLIO\-KR\. For Gemma on P\-FOLIO\-KR, we observe that NL gets the biggest AG increase in S2\. The performance of NL \+ CLIF, which is the best by default, decreases when adding descriptions\. S2 also lowers the performance of abstract notations \(i\.e\. CLINGO, TFLPLUS\)\. With no fine\-tuning in ZS, we can hypothesize that the impact of descriptions varies with model sensitivity to the notation\. Since Gemma is mainly pre\-trained on text data, new notations or combinations \(e\.g\. TFLPLUS, NL \+ CLIF\) might lower model performance when prompted with added descriptions\. For Llama, its pre\-training on distilled data from highly filtered web content makes it less averse to symbolic tokens than models like Gemma\. This is shown by its performance on FOL and FOL\-like notations \(e\.g\. MINIFOL2, CLINGO\) in S1 and S2\. Finally, Phi being pre\-trained on synthetic and logical \(e\.g\. mathematical\) data may explain its performance increase when adding descriptions for abstract notations\.

For FOLIO\-KR, previous experimental results\[[2](https://arxiv.org/html/2608.12374#bib.bib1)\]have shown that Gemma bests Llama and Phi in ZS setting, making it the sole focus of S2 experiments for this dataset\. Results strengthen the observations seen in P\-FOLIO as reasoning improves when adding descriptions on NL\-like notations like CLIF and the combination NL \+ CLIF\. TFLPLUS is a curious case that receives 0 AG in S2 on this larger dataset\. One possible explanation for this behavior might be that the model plateaued out due to the high abstraction level of the notation\.

Table 7:Gemma\-2\-2b\-it ZS Scenario 2 runtime \(in minutes\) on FOLIO\-KR\. Best results in bold, second\-best underlined\.NotationNL \+ CLIFNLCLIFCLINGOCGIFFOLMINIFOL2TFLPLUSRuntime \(min\)3730232527282421

#### 5\.2\.3KR notations show faster reasoning inference\.

Table[7](https://arxiv.org/html/2608.12374#S5.T7)shows the ZS inference runtime of different notations with Gemma\-2 in S2\. The results are limited to Gemma since it is the overall best\-performing model and to FOLIO\-KR since it is 4 times larger than P\-FOLIO\-KR, making runtime differences more pronounced\. Observations show that NL has the longest runtime, making it the most computationally expensive notation\. TFLPLUS is the most compact and fastest syntax\. CLIF appears as a good compromise between runtime and performance based on Table[6](https://arxiv.org/html/2608.12374#S5.T6)\. Combining notations \(e\.g\. NL \+ CLIF\) can increase performance in ZS at the expense of longer runtimes\. These results further support our claim that notation choice depends on multiple factors and should be considered in the design of multi\-stage reasoning pipelines\.

## 6Conclusion

In this work, we introduced the CLGC framework, which converts syllogisms represented in FOL into alternative formal notations, to study of the impact of KR notations on syllogistic reasoning in SLMs\. Our findings show that model size increase performance independently of notation abstraction\. Additionally, combinations of notations like NL \+ CLIF shift reasoning on syllogisms, improving refutation and uncertainty\. Our results also show that providing descriptions of syllogism categories in prompt can improve model performance, but the improvement varies depending on the notation and model used\. We released the FOLIO\-KR and P\-FOLIO\-KR datasets and made the CLGC framework and experiments publicly available\. CLGC demonstrates that small, frugal models can perform well on small reasoning datasets with the right choice of notation\. The dichotomy between natural and abstract notations paves the way for future work on neuro\-symbolic model architectures combining both to carry out complex, multi\-stage reasoning in different training paradigms\.

\{credits\}

#### 6\.0\.1Acknowledgements

This work was supported by the French government through the France 2030 investment plan managed by the National Research Agency \(ANR\), as part of the Initiative of Excellence Université Côte d’Azur \(ANR\-15\-IDEX\-01\), by the 3IA Côte d’Azur \(ANR\-19\-P3IA\-0002\), by the Université Côte d’Azur’s Center for High\-Performance Computing and by the Data ScienceTech Institute\.

#### 6\.0\.2\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]H\. Abi Akl, F\. Gandon, C\. Faron, and P\. Monnin\(2026\)SEF\-clgc at semeval\-2026 task 11: logical notation impact on language model performance\.InSemEval\-2026\-The 20th International Workshop on Semantic Evaluation,Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1),[§3\.1\.3](https://arxiv.org/html/2608.12374#S3.SS1.SSS3.p2.1),[§4](https://arxiv.org/html/2608.12374#S4.p1.1)\.
- \[2\]H\. Abi Akl\(2025\)Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering\.InRules and Reasoning, 9th International Joint Conference, RuleML\+RR 2025,Instabul, Turkey\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1),[§3\.1\.2](https://arxiv.org/html/2608.12374#S3.SS1.SSS2.p1.1),[§3\.2](https://arxiv.org/html/2608.12374#S3.SS2.p1.1),[§4](https://arxiv.org/html/2608.12374#S4.p1.1),[§5\.2\.2](https://arxiv.org/html/2608.12374#S5.SS2.SSS2.p2.1)\.
- \[3\]L\. Bertolazzi, A\. Gatt, and R\. Bernardi\(2024\)A systematic analysis of large language models as soft reasoners: the case of syllogistic inferences\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 13882–13905\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[4\]J\. M\. Castro\-Manzano, L\. I\. Lozano\-Cobos, and P\. O\. Reyes\-Cardenas\(2018\)Programming with term logic\.BRAIN\. Broad Research in Artificial Intelligence and Neuroscience9\(3\),pp\. 22–36\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[5\]M\. K\. Chen, X\. Zhang, and D\. Tao\(2025\)Justlogic: a comprehensive benchmark for evaluating deductive reasoning in large language models\.arXiv preprint arXiv:2501\.14851\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[6\]I\. Dasgupta, A\. K\. Lampinen, S\. C\. Chan, H\. R\. Sheahan, A\. Creswell, D\. Kumaran, J\. L\. McClelland, and F\. Hill\(2022\)Language models show human\-like content effects on reasoning tasks\.arXiv preprint arXiv:2207\.07051\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[7\]T\. Eisape, M\. Tessler, I\. Dasgupta, F\. Sha, S\. Steenkiste, and T\. Linzen\(2024\)A systematic comparison of syllogistic reasoning in humans and language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 8425–8444\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[8\]M\. Gebser, R\. Kaminski, B\. Kaufmann, and T\. Schaub\(2014\)Clingo= asp\+ control: preliminary report\.arXiv preprint arXiv:1405\.3694\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[9\]S\. Han, H\. Schoelkopf, Y\. Zhao, Z\. Qi, M\. Riddell, W\. Zhou, J\. Coady, D\. Peng, Y\. Qiao, L\. Benson,et al\.\(2024\)Folio: natural language reasoning with first\-order logic\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 22017–22031\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.12374#S4.SS1.p1.1),[§4\.2\.1](https://arxiv.org/html/2608.12374#S4.SS2.SSS1.p2.1)\.
- \[10\]S\. Han, A\. Yu, R\. Shen, Z\. Qi, M\. Riddell, W\. Zhou, Y\. Qiao, Y\. Zhao, S\. Yavuz, Y\. Liu,et al\.\(2024\)P\-folio: evaluating and improving logical reasoning with abundant human\-written reasoning chains\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 16553–16565\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.12374#S4.SS1.p1.1),[§4\.2\.1](https://arxiv.org/html/2608.12374#S4.SS2.SSS1.p2.1),[§5\.1\.2](https://arxiv.org/html/2608.12374#S5.SS1.SSS2.p1.1)\.
- \[11\]F\. Hoppe, F\. Ilievski, and J\. Kalo\(2025\)Investigating the robustness of deductive reasoning with large language models\.arXiv preprint arXiv:2502\.04352\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[12\]G\. Kim, M\. Valentino, and A\. Freitas\(2025\)Reasoning circuits in language models: a mechanistic interpretation of syllogistic inference\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 10074–10095\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[13\]Y\. Kwon, D\. Moon, Y\. Oh, and H\. Yoon\(2025\)Logicqa: logical anomaly detection with vision language model generated questions\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 411–432\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[14\]Q\. Lyu, S\. Havaldar, A\. Stein, L\. Zhang, D\. Rao, E\. Wong, M\. Apidianaki, and C\. Callison\-Burch\(2023\)Faithful chain\-of\-thought reasoning\.InProceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 305–329\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[15\]J\. M\. C\. Manzano\(2025\)An intermediate term functor logic\.Argumentos \- Revista de Filosofia11\(22\),pp\. 17–31\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[16\]G\. Maraia, M\. Valentino, F\. M\. Zanzotto, and L\. Ranaldi\(2026\)Abstract activation spaces for content\-invariant reasoning in large language models\.arXiv preprint arXiv:2602\.02462\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[17\]K\. Ozeki, R\. Ando, T\. Morishita, H\. Abe, K\. Mineshima, and M\. Okada\(2024\)Exploring reasoning biases in large language models through syllogism: insights from the neubaroco dataset\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 16063–16077\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[18\]X\. Quan, M\. Valentino, L\. A\. Dennis, and A\. Freitas\(2024\)Verification and refinement of natural language explanations through llm\-symbolic theorem proving\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2933–2958\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1),[§1](https://arxiv.org/html/2608.12374#S1.p2.1)\.
- \[19\]L\. Ranaldi, M\. Valentino, and A\. Freitas\(2025\)Improving chain\-of\-thought reasoning via quasi\-symbolic abstractions\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17222–17240\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[20\]A\. Saparov and H\. He\(2022\)Language models are greedy reasoners: a systematic formal analysis of chain\-of\-thought\.arXiv preprint arXiv:2210\.01240\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[21\]S\. Seals and V\. Shalin\(2024\)Evaluating the deductive competence of large language models \(2023\)\.arXiv preprint arXiv:2309\.05452\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[22\]F\. Sommers and G\. Englebretsen\(2017\)An invitation to formal reasoning: the logic of terms\.Routledge\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[23\]J\. Sowa\(2011\)Introduction to common logic\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[24\]J\. F\. Sowa\(1992\)Conceptual graphs as a universal knowledge representation\.Computers & Mathematics with Applications23\(2\-5\),pp\. 75–93\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[25\]J\. F\. Sowa\(2008\)Conceptual graphs\.Foundations of artificial intelligence3,pp\. 213–237\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[26\]J\. F\. Sowa\(2016\)Conceptual graphs for representing conceptual structures\.Conceptual Structures in Practice,pp\. 119–154\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[27\]O\. Tafjord, B\. Dalvi, and P\. Clark\(2021\)Proofwriter: generating implications, proofs, and abductive statements over natural language\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 3621–3634\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1)\.
- \[28\]M\. Valentino, G\. Kim, D\. Dalal, Z\. Zhao, and A\. Freitas\(2026\)Mitigating content effects on reasoning in language models through fine\-grained activation steering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33314–33322\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[29\]M\. Valentinoet al\.\(2026\)SemEval\-2026 task 11: disentangling content and formal reasoning in large language models\.InProceedings of the 20th International Workshop on Semantic Evaluation \(SemEval\-2026\),Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.
- \[30\]F\. Wanget al\.\(2025\)A comprehensive survey of small language models in the era of large language models: techniques, enhancements, applications, collaboration with llms, and trustworthiness\.ACM Transactions on Intelligent Systems and Technology16\(6\),pp\. 1–87\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2608.12374#S4.SS2.SSS1.p2.1),[§4\.2\.2](https://arxiv.org/html/2608.12374#S4.SS2.SSS2.p1.1)\.
- \[31\]M\. Wysockaet al\.\(2025\)SylloBio\-nli: evaluating large language models on biomedical syllogistic reasoning\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7235–7258\.Cited by:[§1](https://arxiv.org/html/2608.12374#S1.p1.1),[§1](https://arxiv.org/html/2608.12374#S1.p2.1)\.
- \[32\]J\. Xuet al\.\(2024\)Faithful logical reasoning via symbolic chain\-of\-thought\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13326–13365\.Cited by:[§2\.1](https://arxiv.org/html/2608.12374#S2.SS1.p1.1)\.