CNC machine tools and relevant machines have been widely used in various fields, including aerospace, instrument making, medicine, mold processing, etc., wherein machining accuracy will affect the final performance of products. In this paper, firstly, the torsion pendulum method for measuring rotational inertia is introduced, and an accurate measurement device is designed based on the physical principle. Next, by applying the Hilbert transform, a novel time-frequency analysis technique is adopted to discover the mathematical formula of the relationship between the rotational inertia of the test object and its undamped natural frequency. Therefore, a computational method of rotational inertia utilizing Hilbert transform is constructed. Lastly, MATLAB simulation and experiments based on the least squares estimation are conducted to validate the effect of the mechanical compensation method for measuring rotational inertia with the help of Hilbert transform. Experimental results demonstrate that the relative error of the computational algorithm proposed in this paper is less than 0.5%, outperforming the traditional periodic algorithm. Moreover, the temperature compensation algorithm makes a significant contribution to enhancing the accuracy of rotational inertia measurement with varying temperature of the torsion bar.