This paper employs multi-scale modeling techniques integrated with computer vision damage detection to achieve automated identification and precise classification of seismic damage characteristics in masonry structures. The multi-scale modeling approach encompasses micro-unit modeling, multi-level modeling, and composite simulation modeling, effectively capturing the complex failure mechanisms of masonry structures during seismic events. Furthermore, the VGG16 neural network is employed for computer vision-based damage assessment. Transfer learning and data augmentation techniques enhance the model’s universality and performance. Additionally, an AI-based risk assessment metric is established to quantitatively evaluate seismic damage in masonry structures by calculating structural response parameters and vulnerability coefficients. Results demonstrate robust performance across varying seismic intensities, maintaining an accuracy of 0.82 with a standard deviation of 0.025 under severe earthquake conditions, indicating high stability and reliability.