• A complete modeling procedure is applied including the modeling of gear mesh stiffness, transmission error and gear pair. • The analytical gear mesh stiffness shows a good agreement with the published result of finite element approach. • The numerical results of proposed dynamic model match well with the published experimental data. • The effects of several essential gear parameters are investigated. • Deeper properties of gear vibration are revealed to guide the actual design and control of gear dynamics. This study is concerned with the nonlinear frequency response characteristics of a spur gear pair system, in which the modeling of time varying mesh stiffness (TVMS) and static transmission error (STE) is stressed. Firstly, a complete modeling procedure is constructed by adopting existing models of TVMS, STE and gear pair, so that the basic gear features and design parameters are contained in the overall gear model. On this basis, a 1 degree-of-freedom (DOF) spur gear pair model is established including the TVMS, STE and nonlinear backlash. Then, to show the validity and reliability, theoretical solutions of the dynamic model are verified by the published numerical results and experimental data. Besides, the stability of the gear system is investigated analytically to obtain the boundaries that separate stable and unstable regions. Finally, parametric studies are conducted to reveal the effects of several key parameters, such as the contact ratio, spacing error, transmitted load and damping ratio. The results show that the modeling of TVMS and STE should be considered and emphasized to obtain precise predictions of system responses and to reveal in depth properties of gear dynamics. Furthermore, the parametric studies and stability analysis offer useful suggestions for researchers and engineers to achieve desirable design and control of dynamic behaviors of gear system.