All-polymer solar cells (all-PSCs) have attracted immense attention in recent years due to their advantages of tunable absorption spectra and electronic energy levels for both donor and acceptor polymers, as well as their superior thermal and mechanical stability. The exploration of the novel n-type conjugated polymers (CPs), especially based on aromatic diimide (ADI), plays a vital role in the further improvement of power conversion efficiency (PCE) of all-PSCs. Here, recent progress in structure modification of ADIs including naphthalene diimide (NDI), perylene diimide (PDI), and corresponding derivatives is reviewed, and the structure-property relationships of ADI-based CPs are revealed.