This article aims to propose a novel performance optimization and dynamic behavior analysis method for rigid flexible coupled multibody mechanical systems based on dynamic response characteristics. Firstly, a multi-body system dynamics optimization model with singular positions is established using the Gaussian principle, transforming the dynamics problem into an optimization problem of finding the extremum of a function. Secondly, the Artificial Fish Swarm Optimization Algorithm (AFSA) is improved by proposing a hybrid algorithm (IAFSA) that combines intelligent optimization and traditional optimization methods to fully utilize the advantages of both and solve dynamic optimization models. This algorithm enhances the global search capability and convergence speed of the algorithm by adding feasible solutions from traditional unconstrained optimization and the optimal solution from the previous time step as additional fish swarm individuals, and introducing mutation operators. Finally, taking the planar flexible double pendulum system as an example, the effectiveness of the proposed method was verified through simulation experiments. The experimental results show that the IAFSA algorithm has higher accuracy and better optimization ability compared to the traditional AFSA algorithm when reverse calculating the initial state angle of the system, and the fitness function value is significantly reduced. Research has shown that the multi-body system dynamics optimization method based on Gaussian principle and improved artificial fish swarm algorithm proposed in this paper can effectively handle multi-body system dynamics problems with singular positions, and improve the accuracy and efficiency of system performance optimization.