We study experimentally various physical limitations and technical imperfections that lead to damping and finite contrast of optically-driven Rabi oscillations between ground and Rydberg states of a single atom. Finite contrast is due to preparation and detection errors and we show how to model and measure them accurately. Part of these errors originates from the finite lifetime of Rydberg states and we observe its $n^3$-scaling with the principal quantum number $n$. To explain the damping of Rabi oscillations, we use simple numerical models, taking into account independently measured experimental imperfections, and show that the observed damping actually results from the accumulation of several small effects, each at the level of a few percents. We discuss prospects for improving the coherence of ground-Rydberg Rabi oscillations in view of applications in quantum simulation and quantum information processing with arrays of single Rydberg atoms.