Polarization, which represents the vector nature of electromagnetic waves, plays a fundamental role in optics. Fast, simple, and broadband polarization state characterization is required by applications such as polarization communication, polarimetry, and remote sensing. However, conventional polarization detection methods face great difficulty in determining the phase difference between orthogonal polarization states and often require a series of measurements. Here, we demonstrate how polarization-dependent holography enables direct polarization detection in a single measurement. Using a multiplexed Pancharatnam–Berry phase metasurface, we generate orthogonally polarized holograms that partially overlap with a spatially varying phase difference. Both amplitude and phase difference can be read from the holographic image in the circular polarization basis, facilitating the extraction of all Stokes parameters for polarized light. The metahologram detects polarization reliably at several near-infrared to visible wavelengths, and simulations predict broadband operation in the 580–940 nm spectral range. This method enables fast and compact polarization analyzing devices, e.g., for spectroscopy, sensing, and communications.