TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation

# : Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation
Source: [https://arxiv.org/html/2606.11637](https://arxiv.org/html/2606.11637)
Kailin Lyu1,3,Di Wu1,Pengwei Zhang1,Yuhang Zheng2,Yingxin Lai4, Long Xiao1,Kangyi Wu5,Pengna Li5,Chen Gao2,Lianyu Hu6,Weihao Yuan7, Xiaobin Hu2,🖂,Jie Hao1,🖂,Ce Hao3,🖂,Shuicheng Yan2,🖂

1Institute of Automation, Chinese Academy of Sciences 2National University of Singapore,3Zhongguancun Academy,4Xiamen University 5Xi’an Jiaotong University,6Nanyang Technological University,7Nanjing University 🖂Corresponding authors

###### Abstract

Touch is a key modality for embodied agents to understand the physical world\. Although recent work has incorporated tactile signals into language systems for tactile commonsense reasoning, scaling such systems to realistic open\-world settings remains challenging due to two key bottlenecks: \(1\) current tactile reasoning datasets remain limited in format and scale, providing insufficient supervision for reasoning from tactile observations to physical commonsense and hindering the learning of transferable tactile commonsense; \(2\) Tactile signals are inherently redundant and action\-specific, yet existing methods often overlook these properties, resulting in inefficient representations with limited semantic expressiveness\. To address these limitations, we propose TouchThinker, a tactile\-language framework that scales tactile commonsense reasoning to the open world from both data and representation perspectives\. First, we construct TouchThinker\-1M, a million\-scale, multi\-source tactile reasoning dataset covering415objects,8scenarios, and7sensor types, providing a solid data foundation for open\-world generalization\. We further introduce TouchThinker\-Bench, an open\-world benchmark with more realistic and diverse tasks\. Then, we propose action\-aware modeling mechanism to improve tactile representation efficiency and enable efficient reasoning\. Experimental results demonstrate that TouchThinker achieves competitive performance against state\-of\-the\-art models across multiple datasets\. Our code and dataset will be made available at:[https://github\.com/lvkailin0118/TouchThinker](https://github.com/lvkailin0118/TouchThinker)\.

![[Uncaptioned image]](https://arxiv.org/html/2606.11637v1/Figure/touchthinker_logo.png): Scaling Tactile Commonsense Reasoning to the Open World with Large\-scale Data and Action\-aware Representation

Kailin Lyu1,3, Di Wu1, Pengwei Zhang1, Yuhang Zheng2, Yingxin Lai4,Long Xiao1,Kangyi Wu5,Pengna Li5,Chen Gao2,Lianyu Hu6,Weihao Yuan7,Xiaobin Hu2,🖂,Jie Hao1,🖂,Ce Hao3,🖂,Shuicheng Yan2,🖂1Institute of Automation, Chinese Academy of Sciences2National University of Singapore,3Zhongguancun Academy,4Xiamen University5Xi’an Jiaotong University,6Nanyang Technological University,7Nanjing University🖂Corresponding authors

## 1Introduction

Among human senses, touch is fundamental for perceiving and interacting with the physical world\. It provides physical cues that vision alone cannot reliably capture, including material properties, surface texture, and contact state, thereby supporting physical reasoning and action decisions\(Liet al\.,[2024b](https://arxiv.org/html/2606.11637#bib.bib3)\)\. For example, when a person touches a hard grain of rice, they can infer that it is under\-ripe; similarly, when touching a soft sponge, they can deduce that it is suitable for wiping based on the context\. Therefore, incorporating tactile information into commonsense reasoning frameworks and developing tactile reasoning in open\-world scenarios are crucial for advancing embodied intelligence\.

Several recent studies have begun to integrate tactile perception with large language models\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4),[2025](https://arxiv.org/html/2606.11637#bib.bib5); Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7); Chenget al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib29)\), enabling robots to perform tactile understanding tasks under natural language instructions\. Despite these advances, existing approaches still struggle to achieve reliable and robust tactile reasoning in open\-world environments, primarily due to limitations in data and representation\. \(1\)On the data side, existing public tactile reasoning datasets remain limited in both format and scale\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4),[2025](https://arxiv.org/html/2606.11637#bib.bib5); Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), hindering models’ ability to perform tactile reasoning in open\-world environments\. Specifically, most rely on limited predefined attributes and question\-answering templates, lacking causal reasoning supervision from tactile observations to physical properties and further to physical commonsense, which can induce hallucinations\(Ishmamet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib11)\)\. Moreover, these datasets typically cover only one to three sensor types, making it difficult for models to distinguish sensor\-specific representational biases from physical property variations shared across sensors, thereby limiting their ability to learn transferable tactile commonsense and cross\-sensor physical reasoning\. \(2\)On the representation side, existing methods still struggle to effectively model tactile features\. First, compared with visual signals, tactile interaction streams typically contain numerous static and transitional frames, whereas task\-relevant attributes are localized within only a few highly informative segments\(Abad and Ranasinghe,[2020](https://arxiv.org/html/2606.11637#bib.bib14)\), resulting in substantial redundancy\. In addition, tactile signals are inherently action\-specific: pressing primarily reveals hardness, sliding captures friction, and rotation exposes texture\. However, existing methods typically rely on uniform sampling or full\-frame encoding, modeling all frames indiscriminately\. This makes it difficult to suppress noise from low\-information frames and explicitly align action types with question semantics, resulting in inefficient representations with limited semantic expressiveness\.

To address these challenges, we introduce TouchThinker, an open\-world tactile\-language framework that advances both data and representation\. On the data side, we construct TouchThinker\-1M, which integrates multi\-source visuotactile data and applies systematic preprocessing, annotation unification, and question\-answer expansion to provide semantically consistent supervision for tactile reasoning\. On the representation side, we design an action\-aware modeling mechanism that first fuses tactile tokens under question guidance and then performs action\-aware Gaussian MoE modeling to identify query\-relevant action segments and reduce representational redundancy\. Based on it, we further develop a two\-stage tactile\-language training paradigm that aligns tactile evidence with language\-based reasoning in the LLM\. Experiments across multiple benchmarks, including the VTV\-150K\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)and our newly constructed TouchThinker\-Bench, demonstrate that TouchThinker achieves higher reasoning accuracy than state\-of\-the\-art methods and supports reliable open\-world tactile inference\.

In summary, our contributions are as follows:

- •Framework: We proposeTouchThinker, a tactile\-language reasoning framework that aligns tactile cues with task semantics through an action\-aware modeling mechanism, enabling efficient tactile representations\.
- •Datasets: We constructTouchThinker\-1M, a million\-scale, multi\-source dataset that expands tactile data across multiple dimensions to support open\-world tactile reasoning\. We further introduceTouchThinker\-Bench, a systematic benchmark covering diverse tactile sensors and task types for rigorous multi\-dimensional evaluation of open\-world tactile commonsense reasoning\.
- •Practice: TouchThinker outperforms existing tactile\-language models across multiple mainstream datasets and subtasks, while demonstrating more reliable physical reasoning\. These results highlight its potential for robotic interaction in open\-world environments and provide new insights for future research on tactile intelligence\.

## 2Related work

Tactile Sensing and Perception\.In recent years, tactile sensing has evolved from early capacitive arrays, piezoresistive sensors, and magnetic sensors to advanced vision based systems capable of capturing high resolution contact information\(Meriboutet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib12); Liet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib13)\)\. Visuotactile sensors have attracted growing attention because they can record fine grained spatiotemporal deformation on contact surfaces\. Representative systems include DIGIT, GelSight, and Tac3D\(Abad and Ranasinghe,[2020](https://arxiv.org/html/2606.11637#bib.bib14)\)\. Building on these advances, many studies have used visuotactile sensing to infer multidimensional tactile properties, thereby supporting dexterous manipulation tasks such as material classification, grasping, and insertion\(Lyuet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib17); Zhenget al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib16)\)\. Recent research has shifted toward representation learning for tactile data\. UniTouch\(Yanget al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib19)\)and T3\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib20)\)adopt visual self supervised objectives for fine grained feature extraction, while UniT\(Xuet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib21)\)introduces VQGAN\(Esseret al\.,[2021](https://arxiv.org/html/2606.11637#bib.bib22)\)for compact tactile latent space modeling\. To address sensor heterogeneity, several studies employ joint training and alignment strategies to encourage consistent representations across sensors\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7); Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23)\)\. In contrast, we identify representational redundancy in tactile signals and emphasize the action specific nature of tactile information\. Based on this observation, we aim to learn more efficient tactile representations and unlock their potential for complex tactile reasoning\.

Tactile Commonsense Reasoning\.Multimodal large language models jointly model language and visual information, substantially improving cross modal reasoning and reshaping research paradigms\(Yinet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib25); Liet al\.,[2024a](https://arxiv.org/html/2606.11637#bib.bib24); Wuet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib41)\)\. While early work primarily focused on vision language models\(Duet al\.,[2022](https://arxiv.org/html/2606.11637#bib.bib26); Zhanget al\.,[2024a](https://arxiv.org/html/2606.11637#bib.bib27); Denget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib28)\), recent studies have begun to leverage the reasoning and understanding capabilities of large language models to model tactile signals, gradually establishing a touch language modeling paradigm for embodied interaction\. Representative studies such as Octopi\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\), Octopi 1\.5\(Yuet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib5)\), and VTV\-LLM\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)have introduced important methods and benchmark datasets for tactile reasoning\. However, taken together, existing studies remain constrained by limited datasets \(e\.g\., sensor, scale, and formats\), and directly transferring vision\-based methods overlooks the unique properties of tactile perception\. Motivated by this gap, we curate and release TouchThinker\-1M, the largest tactile commonsense reasoning dataset to date to the best of our knowledge\. It substantially surpasses existing benchmarks by scaling the data across multiple dimensions, thereby providing a stronger data foundation\. Coupled with our methodological improvements, TouchThinker enhances the practical applicability and generalization of tactile commonsense reasoning in open\-world scenarios\.

![Refer to caption](https://arxiv.org/html/2606.11637v1/x1.png)Figure 2:Data statistics of TouchThinker\-1M and TouchThinker\-Bench\.\(a\) Scale comparison between TouchThinker\-1M and representative tactile reasoning datasets in terms of sensor type, object coverage, sample size, and task formulation\. \(b\) Distribution of tactile sensor types in TouchThinker\-1M\. \(c\) Distribution of scene categories in TouchThinker\-1M\. \(d\) Semantic taxonomy and category distribution of TouchThinker\-Bench\.
## 3Curating TouchThinker\-1M Dataset

Multi\-Source Tactile Data Collection\.To address the limitations of existing tactile reasoning datasets in format and scale, which constrain models’ reasoning capabilities, we constructTouchThinker\-1M, a large scale multisource visuotactile dataset for open\-world tactile commonsense reasoning\. It systematically integrates mainstream visuotactile data sources from nine datasets, including representative sources such as TacQuad\(Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23)\), PHYSICLEAR\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\), and VTV\-150K\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), comprising approximately1M framesin total\. The incorporation of heterogeneous sources substantially broadens the distribution of tactile signals and provides a foundation for learning generalizable tactile semantic representations\. As shown in Figure[2](https://arxiv.org/html/2606.11637#S2.F2), TouchThinker\-1M significantly improves the scale and diversity of tactile data along five complementary dimensions: sensor type, interaction action, object coverage, sample size, and task formulation\. Dataset details are provided in theAppendix[A\.1\.1](https://arxiv.org/html/2606.11637#A1.SS1.SSS1)\.

Tactile Attribute Annotation and Unification\.Since annotation schemas vary across tactile datasets, directly merging them would lead to an inconsistent semantic space and hinder learning of stable tactile concepts\. To address this, we therefore adapt the VTV\-150K\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)schema into a unified four dimensional tactile attribute space covering Hardness, Protrusion, Elasticity, and Friction\. For data with existing annotations, we map the original labels to this unified schema; for unannotated data, we manually assign attributes based on object appearance, tactile observations, and deformation patterns\. Each sample is independently labeled by multiple annotators, cross verified, and adjudicated when disagreements arise\. Samples with insufficient tactile evidence are excluded, yielding reliable supervision that remains consistent across datasets and sensors\.

Standardized Processing of Tactile Videos\.We standardize all samples into a unified video format to capture tactile interaction dynamics during contact\. For existing tactile videos, we retain valid contact intervals and remove noncontact, noisy, or redundant segments\. For static tactile images, we construct short video sequences by organizing frames according to object category and contact state to capture deformation evolution\. All videos are further processed through contact region cropping, frame rate resampling, interpolation, temporal truncation, and length normalization, yielding clips of approximately 6 to 8 seconds that allow TouchThinker\-1M to encode diverse tactile dynamics in a consistent format\.

Question Answer Format Expansion\.To reduce shallow semantic learning from templated tactile question answering\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4); Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), TouchThinker\-1M additionally introduces two complementary formats: chain\-of\-thought reasoning and open\-ended question answering\. Tactile chain\-of\-thought instances require an intermediate rationale and a final answer using<think\>\.\.\.</think\><answer\>\.\.\.</answer\>, where<think\></think\>records tactile evidence and reasoning paths, and<answer\></answer\>provides attribute judgments or commonsense conclusions\. Open\-ended tactile question answering covers free form description, comparative reasoning, attribute explanation, interaction prediction, and open\-world decision making, enabling more realistic tactile language reasoning\. Construction details are provided in theAppendix[A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2)\.

## 4TouchThinker\-Bench

Existing tactile reasoning benchmarks are often constrained by limited object categories, templated question answering formats, and restricted sensor coverage\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4); Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), making it difficult to assess whether models acquire transferable tactile semantics\. To comprehensively evaluate tactile commonsense reasoning in open\-world scenarios, we introduce TouchThinker\-Bench\. It designed for more realistic open\-world evaluation and incorporates unseen objects, unseen sensors, and diverse task, thereby enabling systematic assessment of tactile understanding, commonsense reasoning, and generalization across sensors\.

Dataset creation and selection\.We first select samples from TouchThinker\-1M using an object\-level 6:1 train\-test split\. To evaluate cross\-sensor generalization, we further incorporate self\-collected data and several additional datasets\(Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23); Zhaoet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib20)\), covering three sensor types unseen during training\. These sensors differ substantially in imaging mechanisms and exhibit intrinsically distinct data characteristics\. Then, we apply data processing procedures similar to those described in Section[3](https://arxiv.org/html/2606.11637#S3)and conduct manual verification, yielding test samples across 200 object categories, with their distribution shown in Figure[2](https://arxiv.org/html/2606.11637#S2.F2)\(d\)\. Further dataset collection processes and dataset details are provided in theAppendix[A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1)\.

Task Taxonomy\.TouchThinker\-Bench comprises three core task categories\.\(1\) Basic tactile property understanding\.This task requires models to identify fundamental tactile attributes, including hardness, roughness, elasticity, and friction\.\(2\) Basic tactile reasoning\.Following prior benchmarks such as Octopi\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\)and VTV\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), this task covers surface feature distinction \(SFD\), surface optimality identification \(SOI\), object sensation correlation \(OSC\), and tactile scenario analysis \(TSA\), enabling stable quantitative comparison\.\(3\) Open\-ended tactile commonsense reasoning\.It requires models to generate free\-form responses for tactile phenomenon description, attribute explanation, and related reasoning tasks, assessing whether they can integrate tactile evidence with real\-world commonsense rather than merely matching fixed templates\. Detailed task definitions and examples are provided in theAppendix[A\.2\.2](https://arxiv.org/html/2606.11637#A1.SS2.SSS2)\.

![Refer to caption](https://arxiv.org/html/2606.11637v1/x2.png)Figure 3:Overview of the proposed TouchThinker framework and its training pipeline\.
## 5Methodology

### 5\.1Tactile Action\-aware Modeling

As discussed in SectionLABEL:intro, tactile signals contain substantial redundancy, with task\-relevant evidence concentrated in only a few action segments\. Modeling all frames indiscriminately amplifies representational redundancy and hinders downstream reasoning\. To address this issue, we propose a tactile action\-aware modeling mechanism that localizes query\-relevant subintervals and extracts meaningful tactile representations, as shown in Figure[3](https://arxiv.org/html/2606.11637#S4.F3)\.

Question\-Guided Token Fusion\.Given a tactile videoV=\{It\}t=1TV=\\\{I\_\{t\}\\\}\_\{t=1\}^\{T\}, we first fine\-tune a base tactile backboneftacf\_\{\\mathrm\{tac\}\}initialized from ViFi\-CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2606.11637#bib.bib6)\)\. Specifically, we attach an attribute classifier and optimize a cross\-entropy loss to predict tactile attributes, including hardness, protrusion, elasticity, and friction\. After training,ftacf\_\{\\mathrm\{tac\}\}is frozen and used to extract frame\-wise tactile features, yielding a tactile token sequenceF=\[f1,…,fT\]∈ℝT×dF=\[f\_\{1\},\\ldots,f\_\{T\}\]\\in\\mathbb\{R\}^\{T\\times d\}\. To incorporate task\-specific priors, we encode the questionqqwith a frozen text encoderftxt​\(⋅\)f\_\{\\mathrm\{txt\}\}\(\\cdot\), obtaining word\-level featuresQw∈ℝL×dtQ\_\{w\}\\in\\mathbb\{R\}^\{L\\times d\_\{t\}\}and a sentence\-level featureqs∈ℝdtq\_\{s\}\\in\\mathbb\{R\}^\{d\_\{t\}\}, which are projected into the tactile space asQ~w=Qw​Ww\\tilde\{Q\}\_\{w\}=Q\_\{w\}W\_\{w\}andq~s=qs​Ws\\tilde\{q\}\_\{s\}=q\_\{s\}W\_\{s\}\. We then use tactile tokens as queries and word\-level question features as keys and values for cross\-modal attention, followed by self\-attention:

Fqa=SelfAttn​\(CrossAttn​\(F,Q~w,Q~w\)\),F\_\{\\mathrm\{qa\}\}=\\mathrm\{SelfAttn\}\\left\(\\mathrm\{CrossAttn\}\\left\(F,\\tilde\{Q\}\_\{w\},\\tilde\{Q\}\_\{w\}\\right\)\\right\),\(1\)The outputFqa∈ℝT×dF\_\{\\mathrm\{qa\}\}\\in\\mathbb\{R\}^\{T\\times d\}provides question\-aware tactile representations that align temporal tactile cues with task semantics while suppressing irrelevant noise \(e\.g\., non\-contact segments\), thereby reducing tactile representational redundancy\.

Action\-aware Gaussian Temporal MoE\.To further focus on reasoning\-relevant tactile evidence, we adapt the MoE framework by introducing a set of tactile experts, each specialized in distinct action patterns\. Rather than discrete experts\(Daiet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib36); Linet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib37)\), we uses context\-dependent routing weights to softly combine their contributions, capturing the continuous dynamics of tactile interactions\. Specially, given the question\-aware tactile tokensFqa=\[fqa,1,…,fqa,T\]F\_\{\\mathrm\{qa\}\}=\[f\_\{\\mathrm\{qa\},1\},\\ldots,f\_\{\\mathrm\{qa\},T\}\], a question\-driven router predicts the expert weightsπ=\{πk\}k=1K\\pi=\\\{\\pi\_\{k\}\\\}\_\{k=1\}^\{K\}from the sentence\-level question representationq~s\\tilde\{q\}\_\{s\}\. Each expert aggregates local tactile evidence through a Gaussian temporal window:

fmoe=∑k=1Kπk​∑t=1Tαk,t​Ek​\(fqa,t\),f\_\{\\mathrm\{moe\}\}=\\sum\_\{k=1\}^\{K\}\\pi\_\{k\}\\sum\_\{t=1\}^\{T\}\\alpha\_\{k,t\}E\_\{k\}\(f\_\{\\mathrm\{qa\},t\}\),\(2\)αk,t=Softmaxt​\(−\(τ^t−μk\)22​σk2\),\\alpha\_\{k,t\}=\\mathrm\{Softmax\}\_\{t\}\\left\(\-\\frac\{\(\\hat\{\\tau\}\_\{t\}\-\\mu\_\{k\}\)^\{2\}\}\{2\\sigma\_\{k\}^\{2\}\}\\right\),\(3\)Here,αk,t\\alpha\_\{k,t\}denotes the temporal attention weight assigned by thekk\-th expert to thett\-th token, andτ^t∈\[0,1\]\\hat\{\\tau\}\_\{t\}\\in\[0,1\]denotes the normalized temporal position\. The parametersμk\\mu\_\{k\}andσk\\sigma\_\{k\}denote the center and width of the temporal window for thekk\-th expert, respectively\. They are generated from theDD\-dimensional question representation through a linear layer, where the predicted center offset is added to an initial temporal center\. This design enables the model to focus on relevant action segments and capture the action\-specific nature of tactile, thereby supporting downstream tactile reasoning\.

### 5\.2Tactile\-Language training paradigm

In the previous section, we introduced a tactile action\-aware mechanism for generating more efficient tactile representations\. We next propose a two\-stage training paradigm that enables the LLM to integrate tactile and linguistic embeddings for multimodal reasoning and response generation\. The training pipeline is illustrated in Figure[3](https://arxiv.org/html/2606.11637#S4.F3)\.

Stage I: Tactile\-Text Alignment\.In this stage, we align tactile encoder outputs with the LLM text embedding space through a tactile\-language adapter, enabling the LLM to interpret tactile inputs\. Trained on TouchThinker\-1M attribute question answering data, the adapter projects tactile representations into textual token embeddings while the LLM\(Wuet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib41)\)remains frozen, focusing optimization on cross\-modal alignment\. Given a tactile videoVVand a text promptpp, the tactile embedding is computed as

EV=Proj​\(Enc​\(V,p\)\),E\_\{V\}=\\mathrm\{Proj\}\(\\mathrm\{Enc\}\(V,p\)\),\(4\)whereEnc\\mathrm\{Enc\}denotes the tactile encoder andProj\\mathrm\{Proj\}denotes the tactile\-language adapter\. The LLM then generates a responseY^=LLMϕ​\(EV,p\)\\hat\{Y\}=\\mathrm\{LLM\}\_\{\\phi\}\(E\_\{V\},p\)based on the tactile embedding and the text prompt\. The model is trained by minimizing the autoregressive cross entropy loss:

ℒce=−𝔼\(V,p,Y\)∼𝒟align​∑i=1Mlog⁡πθ​\(yi∣EV,p,y<i\),\\mathcal\{L\}\_\{\\mathrm\{ce\}\}=\-\\mathbb\{E\}\_\{\(V,p,Y\)\\sim\\mathcal\{D\}\_\{\\mathrm\{align\}\}\}\\sum\_\{i=1\}^\{M\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid E\_\{V\},p,y\_\{<i\}\),\(5\)where𝒟align\\mathcal\{D\}\_\{\\mathrm\{align\}\}is the attribute question answering dataset\.Y=\[y1,…,yM\]Y=\[y\_\{1\},\\ldots,y\_\{M\}\]is the target answer sequence,yiy\_\{i\}is the target token at positionii,y<iy\_\{<i\}denotes the preceding target tokens, andπθ\\pi\_\{\\theta\}denotes the predictive distribution\. We use teacher forcing during training to stabilize tactile\-text alignment\.

Stage II: End\-to\-end Supervised Fine\-Tuning\.In this stage, We perform LoRA\-based supervised fine\-tuning on diverse instruction data, including chain\-of\-thought and open\-ended question answering, to elicit evidence\-driven and logically consistent tactile reasoning\. To preserve stable perceptual representations, we freeze the tactile encoder and update only the tactile\-language adapter and the LoRA parameters inserted into the LLM self\-attention layers\. Each instruction instance is represented as\(V,p,Y\)\(V,p,Y\), whereVVis a tactile video,ppis the text prompt, andYYis the target output\. For chain\-of\-thought instances,YYfollows the structured format<think\>\.\.\.</think\><answer\>\.\.\.</answer\>, where the reasoning traceTTprecedes the final answerAA\. For open\-ended question answering,YYis an open\-format natural\-language response\. Under multi\-format supervision, the model aligns dynamic tactile evidence with language\-based reasoning\. The supervised fine\-tuning objective is defined as

ℒsft=−𝔼\(V,p,Y\)∼𝒟inst​∑i=1Mlog⁡πθ​\(yi∣EV,p,y<i\),\\mathcal\{L\}\_\{\\mathrm\{sft\}\}=\-\\mathbb\{E\}\_\{\(V,p,Y\)\\sim\\mathcal\{D\}\_\{\\mathrm\{inst\}\}\}\\sum\_\{i=1\}^\{M\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid E\_\{V\},p,y\_\{<i\}\),\(6\)where𝒟inst\\mathcal\{D\}\_\{\\mathrm\{inst\}\}denotes the instruction dataset containing attribute question answering, chain\-of\-thought, and open\-ended question answering instances\. This stage strengthens alignment between tactile perception and language reasoning, improving generalization in open\-world tactile question answering\.

## 6Experiments

### 6\.1Settings

Datasets\.We conduct experiments on two benchmarks: the recently recognized VTV\-150K\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)benchmark and our self\-constructed TouchThinker\-Bench\. All test sets contain objects unseen during training, enabling comprehensive evaluation across diverse tactile reasoning tasks\. In addition, TouchThinker\-Bench further includes unseen sensors to assess cross\-sensor generalization\.

Implementation\.All models are trained on four NVIDIA A100\-80G GPUs with a batch size of 16\. Tactile\-language training follows a two\-stage paradigm: Stage I trains the Tactile\-Language Adapter on TouchThinker\-1M attribute question\-answering data using AdamW with a learning rate of2×10−42\\times 10^\{\-4\}; Stage II fine\-tunes the adapter and the LLM LoRA parameters on more complex question\-answering data, including chain\-of\-thought and open\-ended question answering, with the same optimizer and learning rate\. We set the LoRA scaling factor to 256, rank to 128, and maximum training steps to 10,000\. The LLM backbone is Qwen2\.5, including 7B and 14B\(Wuet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib41)\)\. Additional details are provided in theAppendix[A\.3\.1](https://arxiv.org/html/2606.11637#A1.SS3.SSS1)\.

Table 1:Performance comparison of TouchThinker with state\-of\-the\-art methods on the VTV\-150K dataset\.The evaluation covers multiple tasks, with results reported in percentages \(%\); boldface denotes the best performance\. To ensure fair comparison and validate the effectiveness of our method, TouchThinker is trained and evaluated on VTV\-150K following the same protocol\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), and results are reported as the average over multiple runs\.Evaluation Metrics\.For basic tactile property understanding and reasoning, we report subtask\-level accuracy by exactly matching the final conclusions, without considering reasoning processes or semantic equivalence\. For open\-ended questions, we measure semantic similarity with METEOR\(Banerjee and Lavie,[2005](https://arxiv.org/html/2606.11637#bib.bib42)\)and use GPT\-5\(Singhet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib47)\)and DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11637#bib.bib48)\)for multidimensional evaluation\.

### 6\.2Main Results

The main experimental results demonstrate that TouchThinker scales tactile commonsense reasoning to open\-world scenarios with three keyadvantages:\[Adv\.1\]improved tactile property prediction and basic tactile commonsense reasoning,\[Adv\.2\]accurate and coherent open\-ended response generation, and\[Adv\.3\]robust generalization to unseen sensors and objects\.

\[Adv\.1\]To ensure fair comparison and validate the effectiveness of our method, we follow the VTV\-LLM evaluation protocol\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)and quantitatively compare TouchThinker with state\-of\-the\-art models on 500 question\-answer pairs sampled from VTV\-150K\. The results are summarized in Table[1](https://arxiv.org/html/2606.11637#S6.T1)\. On Tactile Feature Analysis task, TouchThinker\-7B outperforms VTV\-LLM\-7B by 5\.2%, 5\.8%, 8\.0%, and 6\.9% in hardness, bumpiness, elasticity, and friction prediction, respectively, demonstrating the effectiveness of its action\-aware tactile representations\. We further evaluate TouchThinker on basic tactile commonsense reasoning tasks, including SFD, SOI, OSC, and TSA\. Compared with VTV\-LLM\-7B, TouchThinker consistently achieves performance gains\. Notably, TouchThinker\-7B also surpasses VTV\-LLM\-14B in overall performance and on most subtasks despite using substantially fewer parameters, highlighting its efficient tactile representation learning and stronger reasoning capability\.

\[Adv\.2\]Prior methods perform reasonably on tactile property prediction and templated reasoning, but their reliance on fixed answer spaces and exact matching limits the evaluation of open\-world tactile explanation and reasoning\. Following the implementation details described in Section[6\.1](https://arxiv.org/html/2606.11637#S6.SS1), we further evaluate TouchThinker on TouchThinker\-Bench for open\-ended tactile question answering, using METEOR\(Banerjee and Lavie,[2005](https://arxiv.org/html/2606.11637#bib.bib42)\)and LLM\-based evaluation\. As shown in Table[2](https://arxiv.org/html/2606.11637#S6.T2), Large\-scale multi\-source data and diversified QA training enable TouchThinker to reduce shallow semantic matching and spurious associations, producing responses that are better grounded in tactile evidence and more logically consistent\. The qualitative comparison in Figure[4](https://arxiv.org/html/2606.11637#S6.F4)further shows that TouchThinker better interprets tactile signals, connects physical properties with real\-world commonsense, and performs open\-world tactile reasoning\.

Table 2:Subtask performance comparison on TouchThinker\-Bench\.We evaluate the models’ open\-ended tactile question\-answering capabilities\. The best results are inbold, and the second\-best ones areunderlined\. TAU: Touch Attribute Understanding\. TIU: Touch Interaction Understanding\. TKU: Touch Knowledge Reasoning\.Table 3:Performance comparison on TouchThinker\-Bench under unseen sensor and object\.Other models suffer substantial performance degradation due to limited generalization, whereas our method maintains robustness\.Table 4:Ablation study of TouchThinker components\.![Refer to caption](https://arxiv.org/html/2606.11637v1/x3.png)Figure 4:Qualitative comparison of responses from TouchThinker and other models\. Errors are highlighted inred, while accurate content is marked inblue\.\[Adv\.3\]We evaluate state\-of\-the\-art Touch\-Language models on TouchThinker\-Bench for tactile property prediction and basic tactile reasoning\. Table[3](https://arxiv.org/html/2606.11637#S6.T3)shows that TouchThinker remains robust to unseen sensors and objects, while Octopi and VTV\-LLM degrade substantially\. This indicates stronger adaptation to heterogeneous sensor imaging mechanisms, signal distributions, and object properties\. This advantage stems from the large\-scale, multi\-source construction of TouchThinker\-1M, which enables models to learn sensor\-invariant tactile semantics rather than sensor\-specific appearance biases, thereby improving generalization to unseen sensing platforms and open\-world objects\.

## 7Ablation Studies

We conduct ablation studies to assess key components of TouchThinker, with additional results provided in theAppendix[A\.4](https://arxiv.org/html/2606.11637#A1.SS4)\.

Impact of the Action\-aware Modeling\.As shown in Table[4](https://arxiv.org/html/2606.11637#S6.T4), removing the action\-aware mechanism leads to consistent performance degradation on both attribute prediction and reasoning tasks, due to the inherent redundancy and action specificity of tactile signals\. In contrast, our action\-aware mechanism optimizes tactile representations\. To intuitively illustrate this effect, we present visualizations in Figure[5](https://arxiv.org/html/2606.11637#S7.F5)\. When the question is related to the protrusion attribute, the action\-aware mechanism dynamically localizes pressing\-related action segments, yielding efficient tactile representations and enhancing tactile understanding\.

Impact of the Training Paradigm\.Table[4](https://arxiv.org/html/2606.11637#S6.T4)validates our two\-stage training paradigm through ablation studies on VTV\-150K\. Removing Stage I leads to misalignment between tactile and textual tokens, reducing the average performance to 58\.3%\. Omitting Stage II removes end\-to\-end fine\-tuning, destabilizes LLM token generation, and causes a sharp performance drop to 53\.3%\. These results confirm that progressive training is crucial for improving model capability\.

![Refer to caption](https://arxiv.org/html/2606.11637v1/x4.png)Figure 5:Visualization of action\-aware temporal weights from tactile Gaussian experts, integrated to focus on question\-relevant frames for accurate reasoning\.
## 8Conclusion

In this work, we present TouchThinker, an open\-world tactile\-language reasoning framework that advances tactile reasoning from both data and methodological perspectives\. First, we construct TouchThinker\-1M, a million\-scale multi\-source visuotactile dataset, and TouchThinker\-Bench, a realistic benchmark for evaluating tactile understanding, commonsense reasoning, and cross\-sensor generalization\. Then, TouchThinker employs a tactile action\-aware modeling mechanism that combines question\-guided fusion with action\-aware Gaussian Temporal MoE to align tactile cues with task semantics, reduce redundancy, and focus on reasoning\-relevant action segments\. A two\-stage tactile\-language training paradigm further aligns tactile evidence with LLM\-based reasoning\. Extensive experiments show that TouchThinker outperforms existing tactile\-language models across benchmarks, with stronger robustness to unseen objects and sensors\. These results highlight the value of scalable tactile data and action\-aware dynamics for reliable open\-world tactile reasoning\.

## 9Limitations

Although TouchThinker demonstrates promising capability in tactile commonsense reasoning, several limitations remain\. First, its supervision is constrained by the diversity of annotated tactile attributes\. The current schema covers common and complementary attributes, including hardness, protrusion, elasticity, friction, and their combinations, while real\-world tactile perception involves additional properties such as malleability and prickliness\. This may limit comprehensive tactile understanding in open\-world settings\. Second, our current implementation focuses mainly on short\-term tactile interactions\. Since TouchThinker\-1M primarily consists of 6 to 7 second contact clips, extending the framework to long\-horizon tactile manipulation remains important for more complex tasks\(Sonet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib46); Tegin and Wikander,[2005](https://arxiv.org/html/2606.11637#bib.bib45)\)\. Third, TouchThinker is evaluated with 7B and 14B LLM backbones, whose computational cost increases with model scale and may hinder deployment on resource\-constrained robotic platforms\. Developing lightweight and accelerated variants is a promising direction for future work\.

## 10Broader Impacts and Ethics Statement

This work advances open\-world tactile commonsense reasoning by aligning language models with tactile perception, enabling embodied agents to acquire more reliable and interpretable physical understanding\. It aligns with broader NLP interests in multimodal grounding, human\-AI interaction, trustworthy evaluation, and responsible data curation\. Potential benefits include improved assistive technologies for visually impaired individuals and applications in manufacturing, quality control, and robotic manipulation\(Liuet al\.,[2020](https://arxiv.org/html/2606.11637#bib.bib43)\)\. We also acknowledge potential ethical risks\. \(1\) Tactile data may encode fine\-grained information about object materials, interaction patterns, or usage contexts, raising privacy and sensitive\-information concerns if improperly collected, stored, or released\. \(2\) Although TouchThinker\-1M integrates multi\-source tactile data, its sensor types, object categories, and attribute space cannot fully capture real\-world variability, which may lead to generalization biases for low\-resource sensors, rare materials, or safety\-critical scenarios\(Zhanget al\.,[2024b](https://arxiv.org/html/2606.11637#bib.bib44)\)\. To mitigate these risks, we emphasize lawful data sources, standardized annotation, cross\-sensor evaluation, and open\-world generalization analysis, and recommend further real\-world validation of safety and feasibility before deployment\.

## References

- Visuotactile sensors with emphasis on gelsight sensor: a review\.IEEE Sensors Journal20\(14\),pp\. 7628–7638\.Cited by:[§1](https://arxiv.org/html/2606.11637#S1.p2.1),[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- S\. Banerjee and A\. Lavie \(2005\)METEOR: an automatic metric for mt evaluation with improved correlation with human judgments\.InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization,pp\. 65–72\.Cited by:[§6\.1](https://arxiv.org/html/2606.11637#S6.SS1.p3.1),[§6\.2](https://arxiv.org/html/2606.11637#S6.SS2.p3.1)\.
- N\. Cheng, J\. Xu, J\. Chen, B\. Fang, and W\. Han \(2026\)STOLA: self\-adaptive touch\-language framework for tactile commonsense reasoning in open\-ended scenarios\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 18198–18206\.Cited by:[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p3.1),[§1](https://arxiv.org/html/2606.11637#S1.p2.1)\.
- H\. Culbertson, J\. J\. L\. Delgado, and K\. J\. Kuchenbecker \(2014\)One hundred data\-driven haptic texture models and open\-source methods for rendering on 3d objects\.In2014 IEEE haptics symposium \(HAPTICS\),pp\. 319–325\.Cited by:[9th item](https://arxiv.org/html/2606.11637#A1.I1.i9.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1)\.
- D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.\(2024\)Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.Cited by:[§5\.1](https://arxiv.org/html/2606.11637#S5.SS1.p3.3)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[Figure 10](https://arxiv.org/html/2606.11637#A1.F10),[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p2.1),[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p3.1),[§A\.2\.2](https://arxiv.org/html/2606.11637#A1.SS2.SSS2.p3.1),[§A\.3\.2](https://arxiv.org/html/2606.11637#A1.SS3.SSS2.p1.1),[§6\.1](https://arxiv.org/html/2606.11637#S6.SS1.p3.1)\.
- Z\. Deng, W\. Ma, Q\. Han, W\. Zhou, X\. Zhu, S\. Wen, and Y\. Xiang \(2025\)Exploring deepseek: a survey on advances, applications, challenges and future directions\.IEEE/CAA Journal of Automatica Sinica12\(5\),pp\. 872–893\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p2.1)\.
- Y\. Du, Z\. Liu, J\. Li, and W\. X\. Zhao \(2022\)A survey of vision\-language pre\-trained models\.arXiv preprint arXiv:2202\.10936\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p2.1)\.
- P\. Esser, R\. Rombach, and B\. Ommer \(2021\)Taming transformers for high\-resolution image synthesis\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 12873–12883\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- R\. Feng, J\. Hu, W\. Xia, T\. Gao, A\. Shen, Y\. Sun, B\. Fang, and D\. Hu \(2025\)Anytouch: learning unified static\-dynamic representation across multiple visuo\-tactile sensors\.arXiv preprint arXiv:2502\.12191\.Cited by:[Figure 9](https://arxiv.org/html/2606.11637#A1.F9),[4th item](https://arxiv.org/html/2606.11637#A1.I1.i4.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p3.1),[§2](https://arxiv.org/html/2606.11637#S2.p1.1),[§3](https://arxiv.org/html/2606.11637#S3.p1.1),[§4](https://arxiv.org/html/2606.11637#S4.p2.1)\.
- R\. Gao, Y\. Dou, H\. Li, T\. Agarwal, J\. Bohg, Y\. Li, L\. Fei\-Fei, and J\. Wu \(2023\)The objectfolder benchmark: multisensory learning with neural and real objects\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 17276–17286\.Cited by:[8th item](https://arxiv.org/html/2606.11637#A1.I1.i8.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1)\.
- C\. Higuera, A\. Sharma, C\. K\. Bodduluri, T\. Fan, P\. Lancaster, M\. Kalakrishnan, M\. Kaess, B\. Boots, M\. Lambeta, T\. Wu,et al\.\(2024\)Sparsh: self\-supervised touch representations for vision\-based tactile sensing\.arXiv preprint arXiv:2410\.24090\.Cited by:[5th item](https://arxiv.org/html/2606.11637#A1.I1.i5.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1)\.
- M\. F\. Ishmam, M\. S\. H\. Shovon, M\. F\. Mridha, and N\. Dey \(2024\)From image to language: a critical analysis of visual question answering \(vqa\) approaches, challenges, and opportunities\.Information Fusion106,pp\. 102270\.Cited by:[§1](https://arxiv.org/html/2606.11637#S1.p2.1)\.
- H\. Li, Y\. Lin, C\. Lu, M\. Yang, E\. Psomopoulou, and N\. F\. Lepora \(2025\)Classification of vision\-based tactile sensors: a review\.IEEE Sensors Journal\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- M\. Li, K\. Chen, Z\. Bi, M\. Liu, X\. Song, Z\. Jiang, T\. Wang, B\. Peng, Q\. Niu, J\. Liu,et al\.\(2024a\)Surveying the mllm landscape: a meta\-review of current surveys\.arXiv preprint arXiv:2409\.18991\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p2.1)\.
- T\. Li, Y\. Yan, C\. Yu, J\. An, Y\. Wang, and G\. Chen \(2024b\)A comprehensive review of robot intelligent grasping based on tactile perception\.Robotics and Computer\-Integrated Manufacturing90,pp\. 102792\.Cited by:[§1](https://arxiv.org/html/2606.11637#S1.p1.1)\.
- B\. Lin, Z\. Tang, Y\. Ye, J\. Huang, J\. Zhang, Y\. Pang, P\. Jin, M\. Ning, J\. Luo, and L\. Yuan \(2026\)Moe\-llava: mixture of experts for large vision\-language models\.IEEE Transactions on Multimedia\.Cited by:[§5\.1](https://arxiv.org/html/2606.11637#S5.SS1.p3.3)\.
- H\. Liu, D\. Guo, F\. Sun, W\. Yang, S\. Furber, and T\. Sun \(2020\)Embodied tactile perception and learning\.Brain Science Advances6\(2\),pp\. 132–158\.Cited by:[§10](https://arxiv.org/html/2606.11637#S10.p1.1)\.
- K\. Lyu, L\. Xiao, J\. Zeng, J\. Dong, X\. Liu, Z\. Zou, H\. Yang, L\. Shu, and J\. Hao \(2026\)TouchFormer: a robust transformer\-based framework for multimodal material perception\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 18496–18504\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- M\. Meribout, N\. A\. Takele, O\. Derege, N\. Rifiki, M\. El Khalil, V\. Tiwari, and J\. Zhong \(2024\)Tactile sensors: a review\.Measurement238,pp\. 115332\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§5\.1](https://arxiv.org/html/2606.11637#S5.SS1.p2.10)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§A\.3\.2](https://arxiv.org/html/2606.11637#A1.SS3.SSS2.p1.1),[§6\.1](https://arxiv.org/html/2606.11637#S6.SS1.p3.1)\.
- C\. Son, J\. Kim, D\. Kang, S\. Park, C\. Ryu, D\. Baek, G\. Jeong, S\. Jeong, S\. Ahn, C\. Lim,et al\.\(2024\)Behavioral biometric optical tactile sensor for instantaneous decoupling of dynamic touch signals in real time\.Nature Communications15\(1\),pp\. 8003\.Cited by:[§9](https://arxiv.org/html/2606.11637#S9.p1.1)\.
- S\. Suresh, H\. Qi, T\. Wu, T\. Fan, L\. Pineda, M\. Lambeta, J\. Malik, M\. Kalakrishnan, R\. Calandra, M\. Kaess,et al\.\(2024\)NeuralFeels with neural fields: visuotactile perception for in\-hand manipulation\.Science Robotics9\(96\),pp\. eadl0628\.Cited by:[7th item](https://arxiv.org/html/2606.11637#A1.I1.i7.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1)\.
- S\. Suresh, Z\. Si, S\. Anderson, M\. Kaess, and M\. Mukadam \(2023\)Midastouch: monte\-carlo inference over distributions across sliding touch\.InConference on Robot Learning,pp\. 319–331\.Cited by:[6th item](https://arxiv.org/html/2606.11637#A1.I1.i6.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1)\.
- J\. Tegin and J\. Wikander \(2005\)Tactile sensing in intelligent robotic manipulation–a review\.Industrial Robot: An International Journal32\(1\),pp\. 64–70\.Cited by:[§9](https://arxiv.org/html/2606.11637#S9.p1.1)\.
- Universal Robots \(2016\)UR5 Technical Specifications\.Note:Technical specifications, Item no\. 110105, EN 09/2016[https://www\.universal\-robots\.com/media/50588/ur5\_en\.pdf](https://www.universal-robots.com/media/50588/ur5_en.pdf)Cited by:[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p3.1)\.
- C\. Wu, J\. Li, J\. Zhou, J\. Lin, K\. Gao, K\. Yan, S\. Yin, S\. Bai, X\. Xu, Y\. Chen,et al\.\(2025\)Qwen\-image technical report\.arXiv preprint arXiv:2508\.02324\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p2.1),[§5\.2](https://arxiv.org/html/2606.11637#S5.SS2.p2.2),[§6\.1](https://arxiv.org/html/2606.11637#S6.SS1.p2.1)\.
- Y\. Xie, M\. Li, S\. Li, X\. Li, G\. Chen, F\. Ma, F\. Yu, and W\. Ding \(2026\)Universal visuo\-tactile video understanding for embodied interaction\.Advances in Neural Information Processing Systems38,pp\. 127864–127883\.Cited by:[1st item](https://arxiv.org/html/2606.11637#A1.I1.i1.p1.1),[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p1.1),[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p2.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1),[§A\.2\.2](https://arxiv.org/html/2606.11637#A1.SS2.SSS2.p2.1),[§1](https://arxiv.org/html/2606.11637#S1.p2.1),[§1](https://arxiv.org/html/2606.11637#S1.p3.1),[§2](https://arxiv.org/html/2606.11637#S2.p1.1),[§2](https://arxiv.org/html/2606.11637#S2.p2.1),[§3](https://arxiv.org/html/2606.11637#S3.p1.1),[§3](https://arxiv.org/html/2606.11637#S3.p2.1),[§3](https://arxiv.org/html/2606.11637#S3.p4.1),[§4](https://arxiv.org/html/2606.11637#S4.p1.1),[§4](https://arxiv.org/html/2606.11637#S4.p3.1),[§6\.1](https://arxiv.org/html/2606.11637#S6.SS1.p1.1),[§6\.2](https://arxiv.org/html/2606.11637#S6.SS2.p2.1),[Table 1](https://arxiv.org/html/2606.11637#S6.T1),[Table 2](https://arxiv.org/html/2606.11637#S6.T2.7.1.5.3.1)\.
- Z\. Xu, R\. Uppuluri, X\. Zhang, C\. Fitch, P\. G\. Crandall, W\. Shou, D\. Wang, and Y\. She \(2025\)UniT: data efficient tactile representation with generalization to unseen objects\.IEEE Robotics and Automation Letters\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- F\. Yang, C\. Feng, Z\. Chen, H\. Park, D\. Wang, Y\. Dou, Z\. Zeng, X\. Chen, R\. Gangopadhyay, A\. Owens,et al\.\(2024\)Binding touch to everything: learning unified multimodal tactile representations\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26340–26353\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.
- F\. Yang, C\. Ma, J\. Zhang, J\. Zhu, W\. Yuan, and A\. Owens \(2022\)Touch and go: learning from human\-collected vision and touch\.arXiv preprint arXiv:2211\.12498\.Cited by:[3rd item](https://arxiv.org/html/2606.11637#A1.I1.i3.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1)\.
- S\. Yin, C\. Fu, S\. Zhao, K\. Li, X\. Sun, T\. Xu, and E\. Chen \(2024\)A survey on multimodal large language models\.National Science Review11\(12\),pp\. nwae403\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p2.1)\.
- S\. Yu, K\. Lin, and H\. Soh \(2025\)Demonstrating the octopi\-1\.5 visual\-tactile\-language model\.arXiv preprint arXiv:2507\.09985\.Cited by:[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p2.1),[§1](https://arxiv.org/html/2606.11637#S1.p2.1),[§2](https://arxiv.org/html/2606.11637#S2.p2.1)\.
- S\. Yu, K\. Lin, A\. Xiao, J\. Duan, and H\. Soh \(2024\)Octopi: object property reasoning with large tactile\-language models\.arXiv preprint arXiv:2405\.02794\.Cited by:[2nd item](https://arxiv.org/html/2606.11637#A1.I1.i2.p1.1),[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p1.1),[§A\.1\.2](https://arxiv.org/html/2606.11637#A1.SS1.SSS2.p2.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p2.1),[§A\.2\.2](https://arxiv.org/html/2606.11637#A1.SS2.SSS2.p2.1),[§1](https://arxiv.org/html/2606.11637#S1.p2.1),[§2](https://arxiv.org/html/2606.11637#S2.p2.1),[§3](https://arxiv.org/html/2606.11637#S3.p1.1),[§3](https://arxiv.org/html/2606.11637#S3.p4.1),[§4](https://arxiv.org/html/2606.11637#S4.p1.1),[§4](https://arxiv.org/html/2606.11637#S4.p3.1),[Table 2](https://arxiv.org/html/2606.11637#S6.T2.7.1.3.1.1),[Table 2](https://arxiv.org/html/2606.11637#S6.T2.7.1.4.2.1)\.
- J\. Zhang, J\. Huang, S\. Jin, and S\. Lu \(2024a\)Vision\-language models for vision tasks: a survey\.IEEE transactions on pattern analysis and machine intelligence46\(8\),pp\. 5625–5644\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p2.1)\.
- Y\. Zhang, Y\. Huang, Y\. Sun, C\. Liu, Z\. Zhao, Z\. Fang, Y\. Wang, H\. Chen, X\. Yang, X\. Wei,et al\.\(2024b\)Multitrust: a comprehensive benchmark towards trustworthy multimodal large language models\.Advances in Neural Information Processing Systems37,pp\. 49279–49383\.Cited by:[§10](https://arxiv.org/html/2606.11637#S10.p1.1)\.
- J\. Zhao, Y\. Ma, L\. Wang, and E\. H\. Adelson \(2024\)Transferable tactile transformers for representation learning across diverse sensors and tasks\.arXiv preprint arXiv:2406\.13640\.Cited by:[Figure 9](https://arxiv.org/html/2606.11637#A1.F9),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p1.1),[§A\.2\.1](https://arxiv.org/html/2606.11637#A1.SS2.SSS1.p3.1),[§2](https://arxiv.org/html/2606.11637#S2.p1.1),[§4](https://arxiv.org/html/2606.11637#S4.p2.1)\.
- Y\. Zheng, S\. Gu, W\. Li, Y\. Zheng, Y\. Zang, S\. Tian, X\. Li, C\. Hao, C\. Gao, S\. Liu,et al\.\(2026\)Omnivta: visuo\-tactile world modeling for contact\-rich robotic manipulation\.arXiv preprint arXiv:2603\.19201\.Cited by:[§2](https://arxiv.org/html/2606.11637#S2.p1.1)\.

## Appendix AAppendix

### A\.1Details of TouchThinker\-1M Dataset

#### A\.1\.1Source Data of TouchThinker\-1M

![Refer to caption](https://arxiv.org/html/2606.11637v1/x5.png)Figure 6:Overview and representative examples of the source datasets in TouchThinker\-1M\.TouchThinker\-1M integrates nine tactile datasets across seven tactile sensing platforms, four major acquisition actions, and over 415 deduplicated objects, comprising 1,001,344 tactile frames in total\. Detailed object categories are provided in Figure[11](https://arxiv.org/html/2606.11637#A1.F11)\. We next provide an overview of the source datasets that constitute TouchThinker\-1M\.

- •VTV150K\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)\. It contributes 150,000 visuo\-tactile video frames from 100 common objects, manually captured with GelSight Mini, DIGIT, and Tac3D sensors through normal pressing, rotational movement, and sliding motion\.
- •PhysiCLEAR\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\)\. It retains 48 everyday objects after duplicate removal and uses a GelSight var\.1 tactile sensor for handheld pressing and rotation, comprising 29,211 tactile frames after curation\.
- •Touch and Go\(Yanget al\.,[2022](https://arxiv.org/html/2606.11637#bib.bib30)\)\. It spans 18 material categories, where human collectors probe real\-world objects with a GelSight17 var\.1 tactile sensor while recording egocentric video, comprising 28,300 tactile frames after curation\.
- •TacQuad\(Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23)\)\. It uses the coarse\-grained spatially aligned subset covering 92 objects, collected with GelSight Mini and DIGIT image\-based tactile sensors through sequential handheld pressing at the same contact location with additional twisting motions, comprising 37,955 tactile frames after curation\.
- •Touch\-Slide\(Higueraet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib33)\)\. It covers 9 toy\-kitchen objects collected as human sliding interaction trajectories with a DIGIT tactile sensor, comprising 12,150 tactile frames after curation\.
- •YCB\-Slide\(Sureshet al\.,[2023](https://arxiv.org/html/2606.11637#bib.bib32)\)\. YCB\-Slide covers 7 YCB objects collected by sliding a handheld DIGIT tactile sensor across fixed object surfaces and recording trajectories, comprising 9,450 tactile frames after curation\.
- •FeelSight\-Real\(Sureshet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib34)\)\. It covers 5 real\-world objects collected with four DIGIT vision\-based tactile sensors mounted on the fingertips of an Allegro hand, using an in\-hand rotation policy to record vision, touch, and proprioception, comprising 21,000 tactile frames after curation\.
- •ObjectFolder\-Real\(Gaoet al\.,[2023](https://arxiv.org/html/2606.11637#bib.bib31)\)\. It covers 70 real\-world household object instances, collected with GelSight17 var\.2 tactile sensors mounted on a Franka Emika Panda arm by contacting selected surface points along their normal directions and recording gel deformation, comprising 686,880 tactile frames after curation\.
- •HaTT\(Culbertsonet al\.,[2014](https://arxiv.org/html/2606.11637#bib.bib35)\)\. It covers 66 material textures using GelSight var\.2 single\-press videos augmented from the original HaTT dataset as tactile data, comprising 26,398 tactile frames after curation\.

#### A\.1\.2Tactilc Instruction Data Synthesis

Template\-based instructions\.Following Octopi\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\)and VTV\-LLM\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), we construct template\-based tactile instructions by instantiating predefined templates with attribute labels and tactile descriptions\. The generated data cover basic tactile property understanding and question\-answering tasks, including tactile feature analysis \(TFA\), surface feature distinction \(SFD\), surface optimality identification \(SOI\), object sensation correlation \(OSC\), and tactile scenario analysis \(TSA\), enabling models to learn tactile\-semantic correspondences across surface properties, object sensations, and interaction contexts\. However, this template\-driven paradigm is constrained by template coverage and expressiveness; its fixed wording and label\-based supervision may encourage shallow correlations rather than grounded tactile reasoning\.

Tactile Chain\-of\-Thought Instructions\.Prior work\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4),[2025](https://arxiv.org/html/2606.11637#bib.bib5); Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)mainly relies on attribute classification or template\-based question answering, yielding discrete labels or brief responses without explicitly modeling contact dynamics or causal dependencies\. This limitation induces shallow tactile\-semantic associations\. To address it, we construct a high\-quality tactile chain\-of\-thought reasoning dataset grounded in intrinsic object\-state changes during dynamic tactile interactions\. The construction pipeline is detailed below\.Stage 1: Data Preparation\. We first curate tactile videos collected under a unified acquisition protocol, retaining key contact and motion segments that reveal object\-property variations\. We then standardize task metadata, including question categories, task instructions, and answer annotations, to provide a consistent training format\.Stage 2: Chain\-of\-Thought Generation\. We reformulate tactile understanding as cross\-modal question answering and attribute\-description generation\. Using predefined prompt templates, we guide a large language model\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11637#bib.bib48)\)to generate reasoning traces grounded in deformation dynamics, contact\-region changes, and geometric or texture cues observed in tactile videos\. Each output is organized as\.\.\.<think\>\.\.\.</think\><answer\>\.\.\.</answer\>, where the reasoning is aligned with key interaction phases and the answer summarizes the inferred physical property\.Stage 3: Manual Filtering\. We conduct consistency checking and manual review to ensure data quality\. Samples are discarded if their reasoning fails to capture key interaction stages or if the intermediate reasoning conflicts with the final answer, thereby preserving logical consistency between chain\-of\-thought supervision and attribute prediction\.Stage 4: Formatting and Data Integration\. Finally, we integrate tactile videos, task annotations, structured prompts, chain\-of\-thought texts, and final answers into a unified training\-ready format\. This standardized pipeline enables supervision for dynamic tactile reasoning and improves tactile understanding and decision reliability in open\-world scenarios\.

![Refer to caption](https://arxiv.org/html/2606.11637v1/x6.png)Figure 7:Example of an open\-ended tactile instruction instance in TouchThinker\-1M, where multi\-granularity tactile context is used to construct grounded conversations about physical properties and touch\-based interaction reasoning\.System Prompt[⬇](data:text/plain;base64,bWVzc2FnZXMgPSBbeyJyb2xlIjogInN5c3RlbSIsICJjb250ZW50IjogZiIiIllvdSBhcmUgYW4gQUkgdGFjdGlsZSBleHBlcnQgaW50ZXJhY3Rpbmcgd2l0aCBhIHNpbmdsZSB0b3VjaGVkIG9iamVjdC4gVGhlIGlucHV0IHByb3ZpZGVzIHRoZSB0YWN0aWxlIGNvbnRleHQgb2YgdGhpcyBvYmplY3QsIHdoaWNoIGNvbnRhaW5zIHR3byBjb21wbGVtZW50YXJ5IGRlc2NyaXB0aW9ucyBvZiB0aGUgc2FtZSBvYmplY3Q6IGEgZGV0YWlsZWQgZGVzY3JpcHRpb24gYW5kIGFuIGF0dHJpYnV0ZSBkZXNjcmlwdGlvbi4gV2hlbiBhbnN3ZXJpbmcgYWxsIHF1ZXN0aW9ucywgdHJlYXQgdGhlIHRhY3RpbGUgY29udGV4dCBhcyB0aGUgaW5mb3JtYXRpb24gcGVyY2VpdmVkIGR1cmluZyBjb250YWN0LgoKRGVzaWduIGEgY29udmVyc2F0aW9uIGJldHdlZW4geW91IGFuZCBhIHBlcnNvbiBhc2tpbmcgYWJvdXQgdGhlIHRvdWNoZWQgb2JqZWN0LiBUaGUgcXVlc3Rpb25zIHNob3VsZCBiZSBkaXZlcnNlLCBhbmQgdGhlIGFuc3dlcnMgbXVzdCBiZSBncm91bmRlZCBvbmx5IGluIHRoZSBnaXZlbiB0YWN0aWxlIGNvbnRleHQuCgpUaGUgY29udmVyc2F0aW9uIHNob3VsZCBpbmNsdWRlIHF1ZXN0aW9ucyBhYm91dCB0aGUgb2JqZWN0J3MgcGh5c2ljYWwgdGFjdGlsZSBwcm9wZXJ0aWVzLCBzdWNoIGFzIGhhcmRuZXNzLCBwcm90cnVzaW9uLCBlbGFzdGljaXR5LCBmcmljdGlvbiwgYW5kIHJlbGF0ZWQgYXNwZWN0cy4gT25seSBhc2sgcXVlc3Rpb25zIHRoYXQgY2FuIGJlIGNsZWFybHkgYW5zd2VyZWQgZnJvbSB0aGUgdGFjdGlsZSBjb250ZXh0IG9yIGFuc3dlcmVkIHJlbGlhYmx5LgoKQWxzbyBpbmNsdWRlIGNvbXBsZXggcXVlc3Rpb25zIHJlbGF0ZWQgdG8gdGhlIHRvdWNoZWQgb2JqZWN0LCBzdWNoIGFzIHF1ZXN0aW9ucyBpbnZvbHZpbmcgdGFjdGlsZSBjb21tb25zZW5zZSBvciBpbnRlcmFjdGlvbiBhbmQgcGVyY2VwdGlvbiBkdXJpbmcgdG91Y2guIFRoZXNlIHF1ZXN0aW9ucyBtYXkgZGlzY3VzcyBncmFzcGFiaWxpdHksIHByZXNzdXJlIGZlZWRiYWNrLCBzdXJmYWNlIGRpc2NyaW1pbmFiaWxpdHksIG1hbmlwdWxhdGlvbiBzdGFiaWxpdHksIGFuZCByZWxhdGVkIGFzcGVjdHMuIERvIG5vdCBhc2sgYWJvdXQgdW5jZXJ0YWluIGRldGFpbHMuIENvbXBsZXggYW5zd2VycyBzaG91bGQgYmUgbW9yZSBkZXRhaWxlZCwgY2xlYXJseSBzdHJ1Y3R1cmVkLCBhbmQgc3VwcG9ydGVkIGJ5IGV4YW1wbGVzIG9yIHJlYXNvbmluZyB3aGVuIG5lY2Vzc2FyeS4iIiJ9XQ==)messages=\[\{"role":"system","content":f"""YouareanAItactileexpertinteractingwithasingletouchedobject\.Theinputprovidesthetactilecontextofthisobject,whichcontainstwocomplementarydescriptionsofthesameobject:adetaileddescriptionandanattributedescription\.Whenansweringallquestions,treatthetactilecontextastheinformationperceivedduringcontact\.Designaconversationbetweenyouandapersonaskingaboutthetouchedobject\.Thequestionsshouldbediverse,andtheanswersmustbegroundedonlyinthegiventactilecontext\.Theconversationshouldincludequestionsabouttheobject’sphysicaltactileproperties,suchashardness,protrusion,elasticity,friction,andrelatedaspects\.Onlyaskquestionsthatcanbeclearlyansweredfromthetactilecontextoransweredreliably\.Alsoincludecomplexquestionsrelatedtothetouchedobject,suchasquestionsinvolvingtactilecommonsenseorinteractionandperceptionduringtouch\.Thesequestionsmaydiscussgraspability,pressurefeedback,surfacediscriminability,manipulationstability,andrelatedaspects\.Donotaskaboutuncertaindetails\.Complexanswersshouldbemoredetailed,clearlystructured,andsupportedbyexamplesorreasoningwhennecessary\."""\}\]Figure 8:Prompt template for generating open\-ended tactile instruction data in TouchThinker\-1M\. The system prompt guides the model to synthesize evidence\-grounded, touch\-centric dialogues from the supplied tactile context, covering low\-level tactile attributes and higher\-level interaction\-aware perceptual reasoning\.Open\-ended Tactile QA Instructions\.To enhance semantic understanding and interaction\-oriented reasoning in open\-ended tactile scenarios, we construct tactile instruction data in an open\-ended question\-answering format, enabling models to learn aligned mappings from tactile context to language responses\. Inspired by Cheng et al\.\(Chenget al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib29)\), we design an automated generation pipeline based on the raw tactile data in TouchThinker\-1M\. This pipeline builds on object\-level tactile data, object identifiers, and manually annotated tactile attribute descriptions, and uses DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11637#bib.bib48)\)to generate instruction\-following samples for tactile interaction\. Specifically, we generate open\-ended tactile question\-answering data from two complementary perspectives: intrinsic tactile attributes, including hardness, protrusion, elasticity, and friction, and interaction\-oriented perceptual experiences, including graspability, pressure feedback, surface discriminability, and manipulation stability\. This design covers both low\-level tactile properties and realistic touch\-based reasoning scenarios\. We first construct an exemplar pool, where each exemplar contains a tactile context and a grounded dialogue, as illustrated in Figure[7](https://arxiv.org/html/2606.11637#A1.F7)\. The tactile context consists of detailed and attribute\-level descriptions, and only question\-answer pairs explicitly supported by the given tactile evidence are retained\. For each target sample, we use its tactile context as the query and retrieve semantically relevant exemplars as few\-shot demonstrations\. Because TouchThinker\-1M organizes tactile samples, object identifiers, and attribute descriptions at the object level, the retrieved demonstrations help align generated responses with the corresponding tactile evidence\. The complete prompt construction is shown in Figure[8](https://arxiv.org/html/2606.11637#A1.F8)\. We then convert model outputs into single\-turn conversations and manually review and correct them, yielding 5,000 touch\-language instruction\-following samples\.

![Refer to caption](https://arxiv.org/html/2606.11637v1/x7.png)Figure 9:Overview and representative examples of TouchThinker\-Bench\.\(a\) The unseen\-sensor split consists of tactile sensors absent from the training set, including DuraGel from TacQuad\(Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23)\), the commercial GelSight Mini from our self\-constructed dataset, and GelSight Var\. 3 from VisGel\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib20)\)\. These sensors exhibit varying degrees of domain shift from the training sensors in imaging mechanism, color distribution, and gel deformation characteristics, enabling evaluation of cross\-sensor generalization\. \(b\) The unseen\-object split consists of objects withheld from training, evaluating whether models can generalize to open\-world objects with unseen tactile appearances and physical properties\.

### A\.2Details of TouchThinker\-Bench

#### A\.2\.1Source Data of TouchThinker\-Bench

TouchThinker\-Bench consists of two components\. The first is derived from the predefined test splits of TouchThinker\-1M and contains objects held out from training, enabling evaluation of model generalization to novel objects\. The second targets cross\-sensor transfer, comprising DuraGel data from the original TacQuad\(Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23)\)test split, whose sensor is absent during training, as well as our self\-constructed dataset and VisGel\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib20)\), collected with the unseen commercial GelSight Mini and GelSight Var\. 3 sensors, respectively\. Overall, TouchThinker\-Bench covers 10 tactile sensors and 82 test objects, spanning both unseen\-object and unseen\-sensor settings to provide a comprehensive evaluation of tactile perception, attribute reasoning, and cross\-sensor generalization\. We provide a detailed description of each component below\.

Cross\-object generalization evaluation\.It consists of multiple existing tactile datasets from the test portion of TouchThinker\-1M, as shown in Figure[9](https://arxiv.org/html/2606.11637#A1.F9)\(b\)\. The PhysiCLEAR\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\)portion is collected with a GelSight var\.1 tactile sensor and contains 5 daily objects in the test set; the VTV\-150K\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\)portion is collected with three tactile sensors, including GelSight Mini, DIGIT, and Tac3D, and contains 5 common objects; the Touch and Go\(Yanget al\.,[2022](https://arxiv.org/html/2606.11637#bib.bib30)\)portion is collected with a GelSight17 var\.1 tactile sensor and contains 18 material surfaces; the YCB\-Slide\(Sureshet al\.,[2023](https://arxiv.org/html/2606.11637#bib.bib32)\)portion is collected with a DIGIT tactile sensor and contains 3 YCB objects; the Touch\-Slide\(Higueraet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib33)\)portion is collected with a DIGIT tactile sensor and contains 3 toy\-kitchen objects; the HaTT\(Culbertsonet al\.,[2014](https://arxiv.org/html/2606.11637#bib.bib35)\)portion is collected with a GelSight var\.2 tactile sensor and contains 9 material textures; the FeelSight\-Real\(Sureshet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib34)\)portion is collected with four DIGIT visuotactile sensors mounted on the fingertips of an Allegro hand and contains 1 real object; and the ObjectFolder\-Real\(Gaoet al\.,[2023](https://arxiv.org/html/2606.11637#bib.bib31)\)portion is collected with GelSight17 var\.2 tactile sensor and contains 9 real household object\.

Cross\-sensor generalization evaluation\.It consists of data collected with three tactile sensors unseen during training, as shown in Figure[9](https://arxiv.org/html/2606.11637#A1.F9)\(a\)\. The TacQuad subset is collected using the DuraGel tactile sensor and contains 15 test objects\(Fenget al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib23)\), while the VisGel subset is collected using the unseen GelSight Var\. 3 tactile sensor\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib20)\)and includes 4 objects\. To further assess performance in real\-world deployment scenarios, we additionally collected tactile data for 10 categories of everyday objects using a UR5 robotic arm\(Universal Robots,[2016](https://arxiv.org/html/2606.11637#bib.bib50)\)equipped with the commercial GelSight Mini sensor\.

#### A\.2\.2Task Types

Basic Tactile Property Understanding\.This task evaluates whether models can identify fundamental tactile attributes, including hardness, roughness, elasticity, and friction\. The evaluation covers both unseen sensors and unseen objects, thereby assessing the generalization ability of tactile representations across sensor domains and object categories\.

Basic Tactile Reasoning\.Following prior benchmarks such as Octopi\(Yuet al\.,[2024](https://arxiv.org/html/2606.11637#bib.bib4)\)and VTV\-LLM\(Xieet al\.,[2026](https://arxiv.org/html/2606.11637#bib.bib7)\), this task includes surface feature distinction \(SFD\), surface optimality identification \(SOI\), object sensation correlation \(OSC\), and tactile scenario analysis \(TSA\)\. These tasks enable standardized quantitative comparison of models’ tactile reasoning capabilities\.

Open\-ended tactile commonsense reasoning\.We construct open\-ended tactile question\-answering samples from TouchThinker\-Bench using tactile images and their category labels\. Specifically, we design a unified prompt template for three task types and use category labels and tactile attribute cues as conditional inputs to DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11637#bib.bib48)\)for generating candidate question\-answer pairs, followed by manual verification\. The benchmark covers Touch Attribute Understanding, Touch Interaction Understanding, and Touch Knowledge Reasoning\. These tasks respectively evaluate models’ ability to recognize basic tactile properties, such as hardness, protrusion, elasticity, and friction; reason about interaction\-related characteristics, such as graspability, pressure feedback, surface discriminability, and manipulation stability; and infer higher\-level tactile knowledge, including material properties, usage patterns, and everyday interaction scenarios, by integrating tactile cues with object categories\. As shown in Figure[10](https://arxiv.org/html/2606.11637#A1.F10), each question is queried independently to avoid unstable formatting and answer\-segmentation errors when generating multiple long\-form responses\. We then parse, deduplicate, and format the model outputs into single\-turn tactile question\-answering samples\. Finally, we conduct manual review and correction to ensure that each question is supported by tactile and category information and that the reference answers are consistent with the corresponding objects’ tactile attributes and interaction characteristics\.

Open\-ended Tactile QA PromptText ContextYou are an AI tactile assistant interacting with a single touched object\. You will understand the touched object from the text prompt, where the object class is<object class\>\. Imagine that you are physically touching this object\.Design conversations between you and a person asking about this object\. The answers should be written in a tone indicating that you are touching the object and answering based on tactile perception\.All answers must be based on the confirmed tactile attribute tendencies and the object class, and should not introduce details that cannot be supported by the given information\.Object Classa microfiber clothTask Specific Prompt Type 1: Touch Attribute UnderstandingInclude questions asking about the typical tactile physical properties of the touched object, including hardness, protrusion, elasticity, and friction\. Design a separate conversation for each tactile attribute\. Ensure that the conversations contain only confirmed information\.Required format:\[
\{"question": "<question 1\>", "answer": "<answer\>"\},
\{"question": "<question 2\>", "answer": "<answer\>"\},
\{"question": "<question 3\>", "answer": "<answer\>"\},
\{"question": "<question 4\>", "answer": "<answer\>"\},
\{"question": "<question 5\>", "answer": "<answer\>"\}
\]Task Specific Prompt Type 2: Touch Interaction UnderstandingInclude questions asking about interaction and perception characteristics during touch, including graspability, pressure feedback, surface discriminability, and manipulation stability\. Randomly select three of these interaction and perception characteristics, and design a separate conversation for each\. Each conversation should include one question and 3–5 corresponding ground\-truth answers\. Ensure that the conversations contain only confirmed information\.Required format:\[
\{"question": "<question 1\>", "answer": "<answer\>"\},
\{"question": "<question 2\>", "answer": "<answer\>"\},
\{"question": "<question 3\>", "answer": "<answer\>"\}
\]Task Specific Prompt Type 3: Touch Knowledge ReasoningInclude complex reasoning questions involving background knowledge related to the touched object, emphasizing the tactile perspective\. Design two conversations\. Each conversation should include one question and 3–5 candidate ground\-truth answers, with each answer being independent and self\-contained\. Ensure that the conversations contain only confirmed information\. Provide more detailed answers for complex questions\. For example, include concrete examples or reasoning steps to make the content more convincing and well\-organized\. Multiple paragraphs may be used if necessary\.Required format:\[
\{"question": "<question 1\>", "answer": "<answer\>"\},
\{"question": "<question 2\>", "answer": "<answer\>"\}
\]Figure 10:Prompt template for open\-ended question answering in TouchThinker\-Bench\. The upper section presents the multimodal input to DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11637#bib.bib48)\), including a textual prompt and its paired visual input\.Table 5:Training configurations for the two\-stage training paradigm\. Stage I: Tactile\-Text Alignment\. Stage II: End\-to\-end Supervised Fine\-Tuning\.

### A\.3Details of Implementation

#### A\.3\.1Configuration and Hyperparameters

We present the detailed training configuration and hyperparameters in the two stages in Table[5](https://arxiv.org/html/2606.11637#A1.T5)\.

#### A\.3\.2GPT\-5 and DeepSeek\-V4 scoring

Since TouchThinker\-Bench is designed for open\-ended and free\-form tactile commonsense reasoning, model responses may vary substantially in wording, level of detail, and reasoning style\. Therefore, conventional n\-gram\-based metrics such as CIDEr and BLEU\-4 are insufficient to fully capture the quality of generated answers\. To provide a more comprehensive evaluation, we report METEOR as a semantic\-similarity\-oriented metric and further employ GPT\-5\(Singhet al\.,[2025](https://arxiv.org/html/2606.11637#bib.bib47)\)and DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11637#bib.bib48)\)as automatic judges for multidimensional response assessment\.

Specifically, each generated response is evaluated from five perspectives: semantic correctness, tactile consistency, commonsense and reasoning plausibility, information completeness, and language quality\. For each dimension, the judge assigns a score from 1 to 10 and provides a brief rationale\. The final score is computed as the average of the five dimension\-level scores:

S=15​∑i=15si,S=\\frac\{1\}\{5\}\\sum\_\{i=1\}^\{5\}s\_\{i\},\(7\)wheresis\_\{i\}denotes the score of theii\-th evaluation dimension\. The evaluation dimensions are defined as follows:

- •Semantic Correctness:Whether the response is semantically consistent with the reference answer and free from factual errors\.
- •Tactile Consistency:Whether the response accurately reflects tactile perception, physical properties, or interaction\-related sensations, and whether it is consistent with the provided tactile information\.
- •Commonsense and Reasoning Plausibility:Whether the response accords with real\-world physical commonsense and presents a reasonable and convincing reasoning process\.
- •Information Completeness:Whether the response covers the key information required by the question and avoids omitting important tactile attributes or explanations\.
- •Language Quality:Whether the response is clear, natural, and fluent, without obvious grammatical or expression errors\.

### A\.4Additional Ablation Studies

Impact of the model scale\.To investigate the impact of model scale on visuotactile understanding, we evaluate LLM backbones of varying sizes and report the performance of TouchThinker with Qwen\-2\.5\-14B, as shown in Table[1](https://arxiv.org/html/2606.11637#S6.T1)\. The results demonstrate consistent improvements across most subtasks as model size increases, with especially significant improvements on reasoning\-oriented tasks such as SFD\.

Impact of Question\-Guided Token Fusion and Gaussian Temporal MoE in the Action\-Aware Modeling Mechanism\.We conduct ablation experiments on VTV\-150K to validate the effectiveness of the two core components in our action\-aware modeling mechanism\. As shown in Table[6](https://arxiv.org/html/2606.11637#A1.T6), both components contribute to tactile reasoning performance\. Removing Question\-Guided Token Fusion prevents the model from incorporating task semantics during tactile feature aggregation, leading to less focused temporal representations and degraded reasoning accuracy\. Removing Gaussian Temporal MoE also leads to performance degradation, indicating that uniform temporal modeling is insufficient for capturing action\-specific tactile evidence\. In contrast, combining Question\-Guided Token Fusion with Gaussian Temporal MoE achieves the best overall performance\. This demonstrates that early question\-aware fusion helps suppress irrelevant tactile segments, while Gaussian Temporal MoE further localizes task\-relevant action intervals, enabling more efficient tactile representations and stronger downstream reasoning\.

Table 6:Ablation study of Question\-Guided Token Fusion and Gaussian Temporal MoE in the action\-aware modeling mechanism\.Object Categories in TouchThinker\-1Mtangerine, carton, playing card, orange, spoon with rice, nectarine, tomato, rubber glove, pen pad, cotton cloth, velvet, sandpaper, chip bag, rubber slipper sole, candle, suede, silk scarf, fascia ball, chalk, clay, avocado, lemon, banana, kiwi, pineapple, plastic bottle, waffle, sponge, plastic basket, balloon, leather glove, building block protrusion, jelly, piano key, blanket, ceramic cup, oven glove, bark, scouring pad, fur, pine cone, ping pong ball, plastic building block, cork, wooden ruler, eraser, velcro, leather wallet, toilet paper roll, shower mat, baseball, golf ball, sticky note, silicone pad, yoga mat, masking tape roll, rubber band, cotton ball, gauze, computer mouse, headphone, face towel, woven watch strap, rubber watch strap, metal watch strap, wooden block, marble, claw, keyboard, remote control button, toothbrush head, vitamin tablet, tennis ball, towel, absorbent cloth, wrist guard, fine bubble film, coarse bubble film, rice, ridge cup, key, screw, circuit board, mold, sponge sheet, wrench, screwdriver handle, aluminum tube, hairpin, steel wool, huamei, candy, grid bag, ridged plastic bottle, scissors, iron ruler, iron clip, tape, injection tube, wire, scissors handle, dishwashing cloth, mesh strainer scoop, toothbrush handle, scissors blade, TV remote back, pillow, disposable water bottle, game controller thumbstick, tissue paper, mop head, rice spatula handle, game controller buttons, hairbrush bristles, leather book cover, bread knife handle, card holder, stress ball, feather duster head, toilet brush handle, potato, paper towel, game controller shoulder buttons, lanyard, insulating holder, basket, rubber slippers, rice spatula scoop, aluminium foil, pen barrel, mesh strainer handle, egg, game controller keypad, bread knife blade, game controller body, bubble wrap, ice block, clothes peg, TSA lock numbers, millet, nylon shirt, feather duster handle, jeans, microfiber cloth, hairbrush handle, toilet brush bristles, hairbrush bristle ends, bath towel, concrete surface, plastic surface, glass surface, wooden surface, metal surface, brick surface, tile surface, leather surface, synthetic fabric surface, rubber surface, paper surface, tree bark surface, grass surface, soil surface, rock surface, gravel surface, sand surface, plant surface, 3D\-printed object, USB connector, vacuum cup, vacuum cup lid, glass jar, bowl edge, bowl handle, metal can, metal can bottom, metal can top, CD, coffee cup bottom edge, cup shell edge, cup shell interior, glass bottle, hot dog, red metal box, metal box edge, plush toy, metal box, mouse scroll wheel, onion, paper cup, peach, pencil box, pill bottle bottom edge, pill bottle cap, plastic bottle cap, plastic box bottom, duck toy beak, duck toy wing, rubber rabbit toy, rubber rabbit ear, rubber ball, shampoo bottle, glove, sponge brush handle, sponge brush, spoon, plastic box, plastic cup, plastic glove, tomato leaf, wooden cup, rubber toy, shuttlecock feather, marble slab, bamboo surface, bicycle basket, bicycle handle, bicycle pedal, bicycle seat, bicycle wheel, stone, block\-patterned carpet, board brush, cable path, chair, desk, door handle, file envelope, glass door, hinge, bamboo leaf, metal post, metal sign, metal block, paper board, plastic chair, post box, tree branch, stone surface, stone pier, stopcock, tube, van surface, wall corner, well cap, wooden stair surface, wall surface, column, bowl, bowl bottom, coffee cup bottom, grape, paper bag, plastic box bottom edge, fiber chair, lock, metal net, metal stair surface, roadblock, toy banana, toy cheese, toy corn, toy lettuce, toy strawberry, toy tomato, toy bread, toy cookie, toy plum, mustard bottle, cleanser bottle, mug, power drill, hammer, sugar box, tomato soup can, salad plate, dinner plate, hair comb, decorative plate, soup ladle, salad fork, frying spatula, 8\-inch skillet, 10\.25\-inch skillet, 10\.5\-inch griddle, dutch oven, dutch oven lid, rinsing cup, hand scoop, shovel toy, round plate, square plate, cutting board, wine glass, drinking cup, portion cup, cake pan, loaf pan, pestle, mortar, sculpture, ladle, spatula, decorative cast, small fork, large fork, soap dish, beer glass, large container, medium container, small container, vase, plate handle, plate, plate base, display stand, drop funnel, container lid, food pan, large flowerpot, small flowerpot, green vase, blue vase, orange vase, large swan, small swan, spoon holder, utensil container, potato masher, skimmer, pasta server, solid turner, slotted turner, green glass, red glass, scoop, box lid, Stanford frisbee, kettlebell, trim removal tool, bell pepper, dice, pear, pepper grinder, Rubik’s cube, ABS plastic, aluminum, artificial grass, athletic shirt fabric, balsa wood, binder, brick, bubble envelope, CD sleeve, canvas, carbon fiber, ceramic, acrylic, coffee filter, cotton, dot paper, EPDM foam, felt, file portfolio, flannel, fleece, floor tile, wooden floor, folder, frosted acrylic, gift box, glitter paper, greeting card, leather back, leather front, MDF, metal foil, metal mesh, metal shelving, foam, wood, nylon bag, nylon mesh, nylon strap, package foam, painted brick, painted wood, green painted wood, pink foam, plastic mesh, pleather, polyethylene foam, portfolio cover, resin carbon fiber, resume paper, rough paper plate, silk, smooth paper plate, stained wood, stone tile, styrofoam, tarp, terra cotta, textured cloth, textured metal, textured paper, textured rubber, tissue paper, vinyl, wax paper, whiteboard\.Figure 11:Object categories in TouchThinker\-1M\. The figure lists object categories after removing articles, dataset sources, and non\-essential descriptive modifiers for compact presentation\.