Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization
Summary
This paper proposes a context-enhanced transformer using formulaic expression desensitization for extracting problem and method sentences from scientific papers, achieving improvements of 3.71% and 2.67% in macro F1 score on two datasets.
View Cached Full Text
Cached at: 06/26/26, 05:16 AM
# Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization Source: [https://arxiv.org/abs/2606.26481](https://arxiv.org/abs/2606.26481) [View PDF](https://arxiv.org/pdf/2606.26481) > Abstract:Billions of scientific papers lead to the need to identify essential parts from the massive text\. Scientific research is an activity from putting forward problems to using methods\. To learn the main idea from scientific papers, we focus on extracting problem and method sentences\. Annotating sentences within scientific papers is labor\-intensive, resulting in small\-scale datasets that limit the amount of information models can learn\. This limited information leads models to rely heavily on specific forms, which in turn reduces their generalization capabilities\. This paper addresses the problems caused by small\-scale datasets from three perspectives: increasing dataset scale, reducing dependence on specific forms, and enriching the information within sentences\. To implement the first two ideas, we introduce the concept of formulaic expression \(FE\) desensitization and propose FE desensitization\-based data augmenters to generate synthetic data and reduce models' reliance on FEs\. For the third idea, we propose a context\-enhanced transformer that utilizes context to measure the importance of words in target sentences and to reduce noise in the context\. Furthermore, this paper conducts experiments using large language model \(LLM\) based in\-context learning \(ICL\) methods\. Quantitative and qualitative experiments demonstrate that our proposed models achieve a higher macro F1 score compared to the baseline models on two scientific paper datasets, with improvements of 3\.71% and 2\.67%, respectively\. The LLM based ICL methods are found to be not suitable for the task of problem and method extraction\. ## Submission history From: Chengzhi Zhang \[[view email](https://arxiv.org/show-email/478a056a/2606.26481)\] **\[v1\]**Thu, 25 Jun 2026 00:42:16 UTC \(1,327 KB\)
Similar Articles
ProtSent: Protein Sentence Transformers
This article introduces ProtSent, a contrastive fine-tuning framework for protein language models that improves embedding quality for downstream tasks like remote homology detection and structural retrieval.
The Context-Ready Transformer
The paper introduces the context-ready transformer, a recurrent architecture that pre-contextualizes tokens before the transformer block, achieving significant inference speedups (e.g., 1.7x on A100) while matching or exceeding standard transformer performance with fewer layers.
On the Expressive Power of Transformers
A survey paper examining the expressive power of transformers as language recognizers, using concepts and methods from circuit complexity to compare them with classical models of computation.
SeDT: Sentence-Transformer Decision-Transformer Conditioning for Multi-Turn Conversation Reliability
The paper introduces SeDT, a training-free inference-time method that improves LLM reliability in multi-turn conversations by annotating conversation history with cumulative relevance scores from three signals, achieving up to +37.7% performance gains on the Lost-in-Conversation benchmark.
Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification
This paper introduces Strategic 16K, a leakage-controlled dataset for document sensitivity classification, and benchmarks classical and transformer-based models, with BERT achieving top performance while addressing label leakage issues.