Outline

Ingegneria Sismica

Ingegneria Sismica

Key Technologies for Robot Autonomous Manipulation Based on Vision-Control Fusion

Author(s): Xiaoyu Xiong1, Guangtie Zhang1
1University of Science and Technology Beijing 100083, Beijing, China
Xiong, Xiaoyu. and Zhang, Guangtie. “Key Technologies for Robot Autonomous Manipulation Based on Vision-Control Fusion.” Ingegneria Sismica Volume 43 Issue 3: 1-18, doi:10.65102/is20261235.

Abstract

 In unstructured environments, autonomous robot manipulation suffers from high visual perception uncertainty, large control delays, and shallow vision-control fusion, resulting in low success rates and poor trajectory accuracy under disturbances. Existing visual servoing, Diffusion Policy, and vision-language-action (VLA) models mostly employ one-way or static fusion, lacking real-time bidirectional interaction. This study proposes a Bidirectional Vision-Control Fusion Framework (BVCFF). An Uncertainty-Aware Adaptive Fusion mechanism (UAAF) dynamically balances vision and control weights via visual entropy and Lyapunov gradients. A Graph Attention Temporal Fusion network (GAT-TF) captures multimodal long-term dependencies. An end-to-end differentiable joint optimization embeds Lyapunov stability into the composite loss for bidirectional error back-propagation. Gazebo simulation experiments and preliminary real-robot validation on a UR5e platform show superior performance: 94.8% grasping success, 87.6% insertion success, 80.3% dynamic success (simulation) and 87.2%, 76.4%, 68.7% (real-robot), 5.3 mm trajectory error, 43 FPS, and 0.92 robustness, outperforming seven benchmarks including Diffusion Policy and OpenVLA-inspired VLA. The deep bidirectional fusion provides an efficient, robust solution for embodied intelligence deployment.

Keywords
Autonomous Robot Operation; Vision Control Fusion; Uncertainty Awareness; Graph Attention Network; End-to-End Optimization; Embodied Intelligence

Related Articles

Liqin Zheng1, Dongrui Qing2, Yan Zhang1
1School of Mathematics and Statistics, Shaan Xi Xue Qian Normal University Xi’an 710100, P.R.China
2School of Marxism, Xi’an University of Finance and Economics Xi’an 710100, P.R.China
Yanan Gao1, Aiqun Peng2, Nina Ma2
1Management School of Anhui Business and Technology College Hefei 230000, Anhui, China
2Economics and Trade School of Anhui Business and Technology College Hefei 230000, Anhui, China
Ya’ning Liu1, Ping Ma1
1School of Teacher Education, Shihezi University, Shihezi, Xinjiang, 832000, China
Yuhui Li1, Zhongliang Gong1
1College of Mechanical and Intelligent Manufacturing, Central South University of Forestry and Technology, Changsha, Hunan, 410004, China
Hanqing Hu1, Chengjin Liu1, Tianmu Tian1
1School of Management Science and Engineering, Beijing Information Science & Technology University, Beijing 100192