This study investigates the effect of different shrinkage rates and residual stresses between the elements of high-strength aluminum alloy during machining, which will affect the performance of high-strength aluminum alloy and cause significant economic losses to the manufacturer. In order to solve this problem, a finite element model of high-strength aluminum alloy laminar cooling heat–force coupling is established in the ABAQUS platform to simulate the thermal field, phase change, and residual stress development during the cooling process. The residual stresses of the high-strength aluminum alloy blank are obtained by the drilling method. The results show that the error between simulation and experimental measurement is between 6.52% and 13.68%, proving that the finite element model established in this study is accurate. It is concluded from further research that the convective heat transfer coefficient is the main influence on the residual stress of quenched aluminum alloy. Thus, this paper suggests that it is feasible to regulate residual stress through the convective heat transfer coefficient, thereby achieving quality control of high-strength aluminum alloy components at the source.