We present TMR-VLA, an end-to-end framework designed for the motion control of tri-leg silicone-based soft robots.
Miniature magnetic robots face a hardware bottleneck where the robot body is too small to integrate onboard sensors or power. This creates a gap between actuation and perception, often requiring human experts to manually adjust magnetic fields based on visual feedback. Our work aims to bridge this gap by enabling autonomous control through a multi-modal system.
๐ง Technical Framework:
โ End-to-End Mapping: The policy translates sequential endoscope images and natural language instructions directly into low-level coil voltage commands.
โ Action Adaptor: We utilized an EndoVLA-initialized backbone with an Action LoRA Adaptor that allows the model to autoregressively emit voltage increments.
โ TrilegMR-Motion Dataset: The model was trained on a new dataset containing 15,793 image-action pairs across 60 episodes.
โ Diverse Locomotion: The system controls five motion primitives: squatting, leg-lifting, rotation, forward movement, and recovery.
๐ฏ Experimental Results:
โ Success Rate: TMR-VLA achieved an average success rate of 74% across tested motion types.
โ Performance: The model outperformed general-purpose multimodal models (such as Qwen2.5-VL and LLaVA-1.6) in both instruction interpretation and action execution.
โ Inference Speed: Real-time control was demonstrated at approximately 2 Hz using an NVIDIA RTX 5090 GPU.
๐ก Significance: This study addresses the challenge of autonomous control in untethered soft robots without increasing their structural complexity. It provides a foundational baseline for intelligent navigation in complex in-vivo environments.
Big news! ๐ Our lab is proud to announce that 6 of our latest papers have been accepted! ๐
We are incredibly proud of the teamโs hard work and innovation. To give each project the spotlight it deserves, we will be sharing details about each breakthrough one by one over the coming days.
Stay tuned for the updates! ๐คโจ
