In this study, a deep learning-based fusion assessment method for wind and solar resource data is proposed to address the problem of poor accuracy of wind and solar resource assessment under complex terrain. The method adopts Kalman filter for the fusion of wind and solar resource reanalysis data, and introduces convolutional network to extract the deep features of multi-source data. Then parallel spatio-temporal axial attention is introduced to learn the long-time dependency and spatial correlation relations. The high precision reconstruction and spatial distribution prediction of scenery variables are realized by the decoder structure, meanwhile, based on the multi-task learning framework, it makes the model optimize the assessment tasks of wind speed and light simultaneously, which improves the assessment efficiency of the model. The experimental results show that the model is able to effectively improve the assessment accuracy of wind and scenery resources in complex terrain, and the average errors of both the angle of attack and the side-slip angle are reduced by more than 60%. In the comparison of SVR and other classical models, this paper’s algorithm has the best overall performance in all assessment tasks, especially in the fusion data set of scenery variables, the error is reduced by 2.72%~7.05% compared with the comparison model. It proves the effectiveness and superiority of this paper’s model in the assessment of complex terrain scenery resources, which can provide reliable technical support for scenery resources planning.