Three-dimensional (3D) displays have been vastly investigated in the past few decades. Recent development of augmented reality (AR) and virtual reality (VR) has further demanded to compress the 3D display system into a compact platform, such as wearable near-eye displays. Holographic optical elements (HOEs) have received widespread attention owing to their lightweight, thin formfactor, and low cost, and thus have been widely deployed in various 3D display systems. In this review article, we first describe the working principle of some 3D techniques used in AR and VR headsets, and then present 3D display systems employing HOEs, and finally analyze how HOEs influence the system design and performance.