Outline

Ingegneria Sismica

Ingegneria Sismica

Key Technologies for Robot Autonomous Manipulation Based on Vision-Control Fusion

Author(s): Xiaoyu Xiong1, Guangtie Zhang1
1University of Science and Technology Beijing 100083, Beijing, China
Xiong, Xiaoyu. and Zhang, Guangtie. “Key Technologies for Robot Autonomous Manipulation Based on Vision-Control Fusion.” Ingegneria Sismica Volume 43 Issue 3: 1-18, doi:10.65102/is20261235.

Abstract

 In unstructured environments, autonomous robot manipulation suffers from high visual perception uncertainty, large control delays, and shallow vision-control fusion, resulting in low success rates and poor trajectory accuracy under disturbances. Existing visual servoing, Diffusion Policy, and vision-language-action (VLA) models mostly employ one-way or static fusion, lacking real-time bidirectional interaction. This study proposes a Bidirectional Vision-Control Fusion Framework (BVCFF). An Uncertainty-Aware Adaptive Fusion mechanism (UAAF) dynamically balances vision and control weights via visual entropy and Lyapunov gradients. A Graph Attention Temporal Fusion network (GAT-TF) captures multimodal long-term dependencies. An end-to-end differentiable joint optimization embeds Lyapunov stability into the composite loss for bidirectional error back-propagation. Gazebo simulation experiments and preliminary real-robot validation on a UR5e platform show superior performance: 94.8% grasping success, 87.6% insertion success, 80.3% dynamic success (simulation) and 87.2%, 76.4%, 68.7% (real-robot), 5.3 mm trajectory error, 43 FPS, and 0.92 robustness, outperforming seven benchmarks including Diffusion Policy and OpenVLA-inspired VLA. The deep bidirectional fusion provides an efficient, robust solution for embodied intelligence deployment.

Keywords
Autonomous Robot Operation; Vision Control Fusion; Uncertainty Awareness; Graph Attention Network; End-to-End Optimization; Embodied Intelligence

Related Articles

Zhihao Jiang1,2, Limi Chen1,2, Jing Yang1
1Hainan Vocational University of Science and Technology, Haikou 571126, China
2Institute for Mathematical Research, Universiti Putra Malaysia, Serdang 43400, Malaysia
Limi Chen1,2, Zhihao Jiang1,2, Jing Yang1
1Hainan Vocational University of Science and Technology, Haikou 571126, China
2Institute for Mathematical Research, Universiti Putra Malaysia, Serdang 43400, Malaysia
Hui Yuan1, Minjie Chai2, Siqing Xu1, Jinsong Li1, Jinwan Zheng1
1Electric Power Research Institute, State Grid Shanxi Electric Power Co., Ltd., Taiyuan, 030001, Shanxi, China
2Jincheng Power Supply Branch, State Grid Shanxi Electric Power Co., Ltd., Jincheng, 048000, Shanxi, China
Yanhan Zhu1,2
1China Academy of Cultural Heritage, Chaoyang District, 100029, Beijing, China
2Beijing University of Civil Engineering and Architecture, Xicheng District, 100044, Beijing, China
Ken Wang1, Jinhan Shu2, Kan Yuan1
1School of Digital Media, Shenzhen Polytechnic University, Shenzhen 518055, Guangdong, China
2Postdoctoral Mobile Station of Journalism and communication, Fudan University, Shanghai 200433, Shanghai, China