Fault diagnosis of gearbox in engineering can effectively improve operational efficiency and reduce maintenance costs. In this paper, a small sample diagnosis method based on improved generative adversarial networks is proposed. Firstly, the Gradient Boosting is used to optimize the iteration strategy of the generator, the deep learning efficiency is optimized by establishing the weak learner. Then, the decision boundary of fault samples is established by the K-Nearest Neighbor algorithm, the distribution of probability space is measured by Mahalanobis distance continuity. Finally, fault classification and diagnosis are achieved by scoring and fusing fault data with two-stream convolutional networks. The effectiveness of the proposed method is verified by comparison and analysis of experiments. The results showed that the proposed method has higher diagnosis accuracy and classification accuracy in the small sample set fault diagnosis of wind turbine gearbox, and also has better performance in fault generation and strengthening.