CRISPR-Cas9 as a programmable genome editing tool is hindered by off-target DNA cleavage<sup>1-4</sup>, and the underlying mechanisms by which Cas9 recognizes mismatches are poorly understood<sup>5-7</sup>. Although Cas9 variants with greater discrimination against mismatches have been designed<sup>8-10</sup>, these suffer from substantially reduced rates of on-target DNA cleavage<sup>5,11</sup>. Here we used kinetics-guided cryo-electron microscopy to determine the structure of Cas9 at different stages of mismatch cleavage. We observed a distinct, linear conformation of the guide RNA-DNA duplex formed in the presence of mismatches, which prevents Cas9 activation. Although the canonical kinked guide RNA-DNA duplex conformation facilitates DNA cleavage, we observe that substrates that contain mismatches distal to the protospacer adjacent motif are stabilized by reorganization of a loop in the RuvC domain. Mutagenesis of mismatch-stabilizing residues reduces off-target DNA cleavage but maintains rapid on-target DNA cleavage. By targeting regions that are exclusively involved in mismatch tolerance, we provide a proof of concept for the design of next-generation high-fidelity Cas9 variants.