Investigating LLM's Problem Solving Capability -- a Study on Statics Questions

arXiv cs.CL Papers
large-language-models problem-solving statics engineering-education multi-step-reasoning diagrams academic-research

Summary

This paper evaluates LLM performance on statics problems, finding that while text-only questions are handled well, accuracy drops with diagrams and multi-step reasoning, suggesting difficulties in applying visual information consistently.

arXiv:2606.26103v1 Announce Type: new Abstract: Large Language Models (LLMs) have rapidly influenced many aspects of society, particularly education, due to their demonstrated ability to complete assignments and examinations across a wide range of subjects. Although prior studies have examined the educational impact of LLMs, much of the existing work relies on public or open problem datasets and lacks topic-specific analysis. In engineering education, especially within mechanical engineering, systematic investigations of LLM performance on specific problem types remain limited. Instead of using traditional methods that directly ask textbook questions to an LLM tool, our study adopts a model distillation process to evaluate LLM capabilities in solving statics problems. By distilling ChatGPT, we extracted 25 text-only statics questions and further constructed two additional datasets by adding diagrams and modifying their numerical values. Experimental results show that while LLMs perform well on text-only statics problems, their accuracy decreases when diagrams are introduced and the problems require multi-step reasoning. Further analysis suggests that this performance drop is not primarily caused by limitations in image recognition, but rather by difficulties in multi-step reasoning and in consistently applying extracted visual information across successive solution stages.
Original Article
View Cached Full Text

Cached at: 06/26/26, 05:13 AM 

# Investigating LLM's Problem Solving Capability -- a Study on Statics Questions
Source: [https://arxiv.org/abs/2606.26103](https://arxiv.org/abs/2606.26103)
[View PDF](https://arxiv.org/pdf/2606.26103)

> Abstract:Large Language Models \(LLMs\) have rapidly influenced many aspects of society, particularly education, due to their demonstrated ability to complete assignments and examinations across a wide range of subjects\. Although prior studies have examined the educational impact of LLMs, much of the existing work relies on public or open problem datasets and lacks topic\-specific analysis\. In engineering education, especially within mechanical engineering, systematic investigations of LLM performance on specific problem types remain limited\. Instead of using traditional methods that directly ask textbook questions to an LLM tool, our study adopts a model distillation process to evaluate LLM capabilities in solving statics problems\. By distilling ChatGPT, we extracted 25 text\-only statics questions and further constructed two additional datasets by adding diagrams and modifying their numerical values\. Experimental results show that while LLMs perform well on text\-only statics problems, their accuracy decreases when diagrams are introduced and the problems require multi\-step reasoning\. Further analysis suggests that this performance drop is not primarily caused by limitations in image recognition, but rather by difficulties in multi\-step reasoning and in consistently applying extracted visual information across successive solution stages\.

## Submission history

From: Hung\-Fu Chang \[[view email](https://arxiv.org/show-email/e2765be9/2606.26103)\] **\[v1\]**Thu, 30 Apr 2026 20:17:09 UTC \(791 KB\)

Similar Articles

Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods

arXiv cs.CL

This study examines how LLMs suggest research methods (datasets, models, metrics) when prompted only with a research question, finding that LLMs exhibit a strong provider bias and propose a much narrower range of methods compared to actual papers, potentially narrowing researchers' methodological search space.