The current study examines the hybrid Rayleigh–Van der Pol–Duffing oscillator (HRVD) with a cubic–quintic nonlinear term and an external excited force. The Poincaré–Lindstedt technique is adapted to attain an approximate bounded solution. A comparison between the approximate solution with the fourth-order Runge–Kutta method (RK4) shows a good matching. In case of the autonomous system, the linearized stability approach is employed to realize the stability performance near fixed points. The phase portraits are plotted to visualize the behavior of HRVD around their fixed points. The multiple scales method, along with a nonlinear integrated positive position feedback (NIPPF) controller, is employed to minimize the vibrations of the excited force. Optimal conditions of the operation system and frequency response curves (FRCs) are discussed at different values of the controller and the system parameters. The system is scrutinized numerically and graphically before and after providing the controller at the primary resonance case. The MATLAB program is employed to simulate the effectiveness of different parameters and the controller on the system. The calculations showed that NIPPF is the best controller. The validations of time history and FRC of the analysis as well as the numerical results are satisfied by making a comparison among them.