To tackle such problems as the failure to consider diverse behaviors of students and improper dynamism of the curriculum system when teaching the topic of Fundamentals of Accounting, this paper presents a hybrid algorithm approach as the approach to optimize the teaching material and curriculum system. Through the integration of Gaussian mixture models and clickstream data analysis, this can be used to determine the characteristics of student learning behaviors and the model parameters are optimized using the Expectation-Maximization (EM) algorithm. According to the ARCS motivation model, the learning design is based on an all-encompassing teaching evaluation system including both formative and summative evaluations. With the help of empirical mode decomposition, students multi-dimensional behavioral time-frequency features were extracted with the use of 2022 Basic Accounting course at H University School of Accounting as an empirical subject. The connection between patterns of behavior and academic achievement was confirmed and compared the results of teaching in experimental and non-experimental classes. Results show that: The hybrid algorithm is capable of identifying the multimodal distribution features of student behavior. The non-experimental class mean (63.15), median (67), and pass rate (0.77) were significantly lower than the respective measures in the experimental class full sample. There was a significant correlation (0.201) between learning performance and formative assessment at the 1% statistical level.