Sustainable and affordable supply of clean, safe, and adequate water is one of the most challenging issues facing the world. Membrane separation technology is one of the most cost-effective and widely applied technologies for water purification. Polymeric membranes such as cellulose-based (CA) membranes and thin-film composite (TFC) membranes have dominated the industry since 1980. Although further development of polymeric membranes for better performance is laborious, the research findings and sustained progress in inorganic membrane development have grown fast and solve some remaining problems. In addition to conventional ceramic metal oxide membranes, membranes prepared by graphene oxide (GO), carbon nanotubes (CNTs), and mixed matrix materials (MMMs) have attracted enormous attention due to their desirable properties such as tunable pore structure, excellent chemical, mechanical, and thermal tolerance, good salt rejection and/or high water permeability. This review provides insight into synthesis approaches and structural properties of recent reverse osmosis (RO) and nanofiltration (NF) membranes which are used to retain dissolved species such as heavy metals, electrolytes, and inorganic salts in various aqueous solutions. A specific focus has been placed on introducing and comparing water purification performance of different classes of polymeric and ceramic membranes in related water treatment industries. Furthermore, the development challenges and research opportunities of organic and inorganic membranes are discussed and the further perspectives are analyzed.