Visual–tactile action decisions
GPT-as-Tactile-Policy
GPT-6-Astra Understands Touch and Outperforms Vision–Tactile–Language–Action Models
Paper available on arXiv. Code coming soon.
Abstract
General-purpose multimodal models have shown promise as direct robot action policies, but their physical understanding remains underexplored. Taking contact-rich manipulation as a typical setting, we investigate the physical understanding of these models and their ability to use this understanding to guide subsequent control actions. Contact-rich manipulation requires interpreting tactile feedback and adjusting actions after obstruction. Existing learned tactile policies acquire these capabilities through specialized pretraining and per-task optimization, with substantial data and training costs. We present GPT-as-Tactile-Policy, which summarizes expert demonstrations into task guidance and tactile references for contact assessment and corrective action. GPT-6-Astra combines this context with multimodal observations and interaction history to generate control commands and adjust actions through contact feedback. Without additional robotics-specific pretraining or per-task policy fine-tuning, our method achieves 93.8% mean success across five UniVTAC tasks. Overall performance is comparable to state-of-the-art policies, with higher success on selected tasks. On Insert Hole, tactile feedback improves success from 31% to 99%. Execution analysis further demonstrates contact-guided probing and correction, supporting manipulation progress through explicit expert experience and online feedback.
Overview
Project video
Five tasks.
Physical interaction throughout.
We evaluate five of the eight manipulation tasks in the UniVTAC benchmark.
Task success rate
Success rate (%) · Ours: 100 test episodes per taskResults follow the manuscript comparison table. Mean is the equal-weight average over the five tasks. The visual baseline π₀.₅ and the tactile-capable baselines use different decision mechanisms and training procedures.
Without additional robotics-specific pretraining or per-task policy fine-tuning, our policy achieves 93.8% mean success, compared with 96.2% for N₀-VTLA and 84.6% for N₀-TWAM. It matches N₀-TWAM on Insert Hole and achieves the highest Lift Can success rate among the compared methods (95%).
The task suite
Explore the video gallery ↗Successful simulation recordings for all five evaluated tasks.
Insert Hole:
Probe, correct, and advance.
Eight stages of one complete episode. Choose a stage to watch.
Decision details — pause to read
Arrows show matched marker changes between decision observations, at a fixed 3× display gain. Evidence stays fixed while its action executes. A large change can accompany loading or unloading: read its direction, the reference-RMS change, and the recorded action context together. Stage labels summarize the action log.
What changes
when touch is available?
Tactile ablation on Insert Hole, using the same 100 test configurations.
+68 percentage points with tactile feedback
Matched test conditions.
Different contact evidence.
The tactile-free variant removes tactile observations and tactile references, while retaining vision, proprioception, and the action interface. Each configuration receives a fixed Task Skill matched to its available observations.
Initialization, low-level controllers, and execution budgets are identical across the paired configurations. The comparison evaluates tactile information together with its corresponding task guidance.
With and without tactile input
With tactile input
Using tactile changes to assess contact, the policy unloads, tests alternative alignments, and advances along a revised axis to complete insertion.
Without tactile input
The policy retries shallow insertion using visual cues, then backs off and maintains its grasped pose when entry remains uncertain. Insertion is not completed.