Unmanned aerial vehicle (UAV) images have gained extensive attention in varying fields, and the Structure from Motion (SfM) technique has become the gold standard for aerial triangulation of UAV images. With increasing data volume caused by the use of multi-view and high-resolution imaging systems and the enhancement of UAV platform’s endurance, the capability for orientation of large-scale UAV images is becoming a prominent and necessary feature for SfM-based solutions. A classical SfM pipeline consists of three major steps, i.e., (i) feature extraction for an individual image, (ii) feature matching for each image pair, and (iii) parameter solving based on iterative bundle adjustment. Most of the time costs are consumed in the second and third steps. This can be explained from three main aspects. First, for feature matching the large number of images and high overlapping degrees cause high combinational complexity of match pairs. Second, the efficiency of commonly utilized techniques for outlier removal would be seriously degenerated because of high outlier ratios of initial matches. Third, for parameter solving of camera poses and scene structures, the iterative execution of bundle adjustment (BA) leads to high computational costs in the incremental SfM workflow. Thus, this paper gives a systematic survey of the state-of-the-art for match pair selection from both ordered and unordered datasets, for outlier removal of initial matches dominated by outliers, and for efficiency improvement of BA, and conducts an experimental evaluation for six well-known SfM-based software packages on UAV image orientation.