In order to solve the problems of strong subjectivity of color collocation, insufficient efficiency of spatial layout optimization and unclear coordination mechanism of the two in modern painting art creation, this paper constructed an optimization method of modern painting creation based on numerical optimization algorithm. Through digital image processing, color space transformation, saliency detection and region segmentation technology, the digital representation of comprehensive color features and the parametric description of spatial layout structure are realized, and a multi-objective optimization model is established with the joint constraints of color harmony, composition balance, coupling consistency and style preservation. The experimental results show that the scores of comprehensive color harmony, light contrast and saturation balance of the proposed method reach 0.91, 0.87 and 0.89 respectively, and the comprehensive color score reaches 0.90. The score of layout balance, spatial rhythm and whitespace rationality reaches 0.92, 0.89 and 0.88, respectively, and the score of comprehensive layout reaches 0.90. The comprehensive score of the complete model reached 0.91. The results show that the proposed method can effectively improve the comprehensive color unity, composition stability and visual order in modern painting creation, which provides a feasible path for computer-aided art creation and intelligent painting optimization.