Aiming at the problem of improving university English listening and speaking ability, this paper designs an English learning system based on deep learning and artificial intelligence technology, which is used as the kernel of the English teaching model to explore the teaching effectiveness. The system assists students in listening and speaking ability training in the form of intelligent conversation, embedded with the improved EMD-FD speech signal feature extraction algorithm and the multi-parametric English pronunciation quality evaluation model (MPEPQE). The results show that the MFCC extraction algorithm combined with the EMD-FD extraction algorithm is able to extract the features of the high-frequency region of the speech signal more completely than the single MFCC extraction algorithm, which in turn improves the speech recognition rate. Meanwhile, compared with the CNN+LSTM scoring model, the MPEPQE scoring model shows better fitting effect and adaptive ability, and its three index values of Pearson correlation coefficient, accuracy and average score difference are 0.706, 84.52% and 0.613 respectively, which all achieve better results. In addition, a teaching model based on the AI English learning system was constructed, the use of which significantly improved students’ English listening and speaking skills compared to the traditional teaching model (P<0.05). This study makes some necessary attempts and explorations for the full-scale promotion of the use of AI-enabled university English learning system in order to effectively use it to improve students’ listening and speaking abilities.