Plant secondary metabolites (SMs) are not only a useful array of natural products but also an important part of plant defense system against pathogenic attacks and environmental stresses. With remarkable biological activities, plant SMs are increasingly used as medicine ingredients and food additives for therapeutic, aromatic and culinary purposes. Various genetic, ontogenic, morphogenetic and environmental factors can influence the biosynthesis and accumulation of SMs. According to the literature reports, for example, SMs accumulation is strongly dependent on a variety of environmental factors such as light, temperature, soil water, soil fertility and salinity, and for most plants, a change in an individual factor may alter the content of SMs even if other factors remain constant. Here, we review with emphasis how each of single factors to affect the accumulation of plant secondary metabolites, and conduct a comparative analysis of relevant natural products in the stressed and unstressed plants. Expectantly, this documentary review will outline a general picture of environmental factors responsible for fluctuation in plant SMs, provide a practical way to obtain consistent quality and high quantity of bioactive compounds in vegetation, and present some suggestions for future research and development.