Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation

# Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation

Source: https://arxiv.org/html/2505.13792

Siddhant Bhambri, Upasana Biswas, Subbarao Kambhampati
School of Computing & AI, Arizona State University
{siddhantbhambri, ubiswas2, rao}@asu.edu

## Abstract

Recent advances in reasoning-oriented Large Language Models (LLMs) have been driven by the introduction of Chain-of-Thought (CoT) traces, where models generate intermediate reasoning steps before producing an answer. These traces, as in DeepSeek R1, are not only used to guide model inference but also serve as supervision signals for Knowledge Distillation (KD) to improve smaller models. A prevailing but under-examined implicit assumption is that these CoT traces when emitted at inference time are both semantically correct and interpretable for end-users. While there are reasons to believe that these intermediate tokens help improve solution accuracy, in this work, we question their validity (semantic correctness) and interpretability to the end user. To isolate the effect of trace semantics, we design experiments in the Question Answering (QA) domain using a rule-based problem decomposition method. This enables us to create Supervised Fine-Tuning (SFT) datasets for LLMs where each QA problem is paired with either verifiably correct or incorrect CoT traces, while always providing the correct final solution. Trace correctness at inference time is then evaluated by checking the accuracy of every sub-step in decomposed reasoning chains. To assess end-user interpretability, we finetune LLMs with three additional types of CoT traces: R1 traces, R1 trace summaries, and post-hoc explanations of R1 traces. We further conduct a human-subject study with 100 participants asking them to rate the interpretability of each trace type on a standardized Likert scale. Our experiments reveal two key findings—(1) CoT trace correctness is not reliably correlated with the model's generation of correct final answers: correct traces led to correct solutions only for 28% of test-set problems while incorrect traces don't necessarily degrade solution accuracy. (2) In end-user interpretability studies, fine-tuning on verbose R1 traces produced the best model performance but these traces were rated as least interpretable by users, scoring on average 3.39 for interpretability and 4.59 for cognitive load metrics on a 5-point Likert scale. In contrast, the decomposed traces that are judged significantly more interpretable don't lead to comparable solution accuracy. Together, these findings challenge the assumption in question, suggesting that researchers and practitioners should decouple model supervision objectives from end-user-facing trace design.

Project Webpage (https://sbhambr1.github.io/trace-check-qa-paper/)
Code (https://github.com/sbhambr1/Trace_Check_QA)
Datasets (https://hf.co/collections/sbhambr1/trace-check-qa-datasets)

## 1 Introduction

Reasoning with intermediate Chain-of-Thought (CoT)-style traces (step-by-step outputs that models produce prior to an answer) has become one of the defining strategies for improving the performance of Large Language Models (LLMs) across diverse problem types, as exemplified by approaches like DeepSeek R1. While models such as DeepSeek R1 often produce extremely verbose unstructured responses even for simple problems, these reasoning traces are utilized both as inference aids and supervision signals in Knowledge Distillation (KD) when Supervised Fine-Tuning (SFT) smaller LLMs for enhanced task performance.

A common but often implicit assumption behind these CoT traces is that they are semantically correct and interpretable for end-users. Training with these traces is done primarily to improve LLM performance on a given task, but fine-tuning objectives rarely require these traces to be semantically correct or interpretable. In this work, we challenge this assumption and ask: "Must CoT reasoning traces be semantically correct and interpretable to end-users for enhancing LLM task performance?"

To address this, we focus our experiments on the Question Answering (QA) domain, where end-users regularly interact with both intermediate traces and final outputs (e.g., ChatGPT, Perplexity, Copilot, Gemini). Faithfulness of reasoning traces is especially critical in these interactive settings where unverifiable traces can lead to loss of user trust, misinformation, errors in model outputs, and perpetuation of biases among other negative consequences.

To assess the trade-offs between semantic correctness of the traces and LLM performance, we design an experimental setting where both final solutions and intermediate traces can be independently evaluated. Specifically, we employ a rule-based problem decomposition technique to break QA tasks into structured sub-problems. Next, we generate SFT datasets pairing questions with either verifiably correct or verifiably incorrect reasoning traces (while always including the correct answer). At inference time, this allows us to verify the correctness of both the final solution and the intermediate traces generated by the distilled model.

To assess the trade-offs between end-user interpretability and LLM performance, we fine-tune models on different types of reasoning traces: DeepSeek R1 traces (verbose CoT outputs), LLM (GPT-4o-mini)-generated summaries of R1 traces (end-user facing summarizations), LLM (GPT-4o-mini)-generated post-hoc explanations (natural language explanations of R1 traces), and verifiably correct traces discussed above. In parallel, we conduct a human-subject study with 100 participants (hired on Prolific), split into four sets of 25. Each group was asked to judge the interpretability of the trace types using a Likert Scale measuring predictability, comprehensibility, and faithfulness attributes.

Our experiments reveal two key findings:

(1) **Correctness of CoT traces is not reliably correlated with LLMs producing correct final answers**: correct traces led to correct solutions only for 28% of test-set problems, while incorrect traces did not consistently degrade answer accuracy.

(2) **End-user interpretability of CoT traces is not reliably correlated with LLMs producing correct final answers**: fine-tuning on verbose DeepSeek R1 traces led to the strongest task performance, yet users rated these traces as least interpretable, scoring on average 3.39 for interpretability and 4.59 for cognitive load metrics on a 5-point Likert scale.

These results highlight that semantic correctness and human interpretability of reasoning traces can in fact be a liability from the perspective of LLM's task performance, challenging assumptions in current LLM supervision practices. The paper is organized as follows: §2 reviews prior work on Large Language & Reasoning Models, Knowledge Distillation, and CoT Trace Interpretability. §3 presents our problem setup, rule-based decomposition for Open-Book QA, and dataset construction for distilling LLMs with correct and incorrect intermediate traces. §4 describes the SFT experiments and human-subject studies, and §5 analyzes results and key insights. We conclude the work in §6. The Appendix includes details on datasets and prompts in App. A, followed by user study and additional results in App. B.

## 2 Related Work

### 2.1 Large Reasoning Models & CoT Traces

Large Language Models (LLMs) have shown remarkable performance on a wide variety of natural language tasks in question answering, text generation, summarization, and translation, to name a few. Recent advances in post-training techniques have led to the rise of Large Reasoning Models (LRMs) such as DeepSeek R1, Google Gemini 2.5, Microsoft Phi-4-reasoning, etc. These reasoning models produce a set of intermediate tokens, commonly referred to as 'reasoning' traces, followed by the final solution. While LRMs have shown significant improvement in final solution accuracy on reasoning tasks over standard LLMs, their intermediate traces are meandering and verbose, making it hard to evaluate their trace validity and end-user interpretability.

### 2.2 Knowledge Distillation

While Small Language Models (SLMs) offer a computationally efficient alternative to LLMs and LRMs, they are not robust to prompt augmentations (such as Chain-of-Thought) or steerable using in-context examples used in few-shot prompt settings. Knowledge Distillation is a well-studied approach used for fine-tuning these SLMs (student) via the outputs of a larger model (teacher). With LRMs generating both an intermediate trace and the final solution, SLMs are also distilled to replicate this output. However, the lack of structured intermediate trace outputs makes the validity of the traces hard to evaluate. This problem is exacerbated for end-user settings such as in Question Answering (QA) domains, where user interactions involve exposure to both intermediate traces and final outputs.

### 2.3 Interpretability of CoT Traces

Some recent works have argued for making these CoT traces more interpretable to the end-user, i.e., improve their faithfulness for the end user, as they are believed to serve as the LLM's explanation to generate the final solution. On the other hand, there has also been work showcasing why these traces are not explainable to the end user. Both sides of this argument stem from the assumption that these traces are indeed meant to be useful and interpretable for the end user and not just for the LLM to improve its final solution performance on a given task. We specifically challenge this assumption and show the disconnect between the use of CoT traces for the LLM (as a training signal in SFT) and the use of CoT traces for the end user (as an interpretable reason behind the model's final solution).

## 3 Knowledge Distillation using Problem Decomposition

This section describes our rule-based problem decomposition method for breaking complex Open Book QA tasks into verifiable sub-problems (§3.1) and explains how we use it to generate structured intermediate traces for SLM distillation (§3.2).

### 3.1 Rule-based Problem Decomposition

In the context of Open Book QA, consider the example shown in Figure 1, which consists of a text passage (or a set of facts) and a question involving temporal reasoning between the queried problem and the facts present in the provided text. Answering this reasoning question involves identifying the relevant fact from the text which satisfies the temporal relation asked in the problem. In this case, the queried fact refers to "Morus Hasratyan works for Haigazian University from 1965 to 1966." The temporal relation queried in the problem is 'during' and thus, the relevant fact that answers the query is "Morus Hasratyan works for History Museum of Armenia from 1964 to 1975." Hence, the final answer is 'History Museum of Armenia'.

From this example, we see that the complex Open Book QA problem can be decomposed into: 1) a **Classification step** determining the type of question asked ('during' temporal relation in this case), and 2) an **Information Retrieval (IR) step** to determine the relevant part of text that can answer the query. Therefore, we utilize these two steps to decompose the Open Book QA problems that allow us to construct structured intermediate traces for evaluation.

### 3.2 Intermediate Trace Generation for SFT

Given the outputs of the sub-problems obtained by decomposing the original query as shown in Figure 1, we generate the intermediate traces in an automated way which consists of the Classification step describing the type of the question posed in the query, and the IR step showing the relevant fact in the text that can help answer the query. We construct a dataset using these Input-Trace-Output tuples that can be utilized to SFT the Small Language Models. Note that by constructing the intermediate trace using these two steps, we can then evaluate the