Autism Spectrum Disorder (ASD) is one of the most prevalent neurodevelopmental disorders, affecting an estimated 1 in 59 children. ASD is highly genetically heterogeneous and may be caused by both inheritable and <i>de novo</i> gene variations. In the past decade, hundreds of genes have been identified that contribute to the serious deficits in communication, social cognition, and behavior that patients often experience. However, these only account for 10-20% of ASD cases, and patients with similar pathogenic variants may be diagnosed on very different levels of the spectrum. In this review, we will describe the genetic landscape of ASD and discuss how genetic modifiers such as copy number variation, single nucleotide polymorphisms, and epigenetic alterations likely play a key role in modulating the phenotypic spectrum of ASD patients. We also consider how genetic modifiers can alter convergent signaling pathways and lead to impaired neural circuitry formation. Lastly, we review sex-linked modifiers and clinical implications. Further understanding of these mechanisms is crucial for both comprehending ASD and for developing novel therapies.