The current bridge maintenance practice generally involves manual visual inspection, which is highly subjective and unreliable. A technique that can automatically detect defects, for example, surface cracks, is essential so that early warnings can be triggered to prevent disaster due to structural failure. In this study, to permit automatic identification of concrete cracks, an ad-hoc faster region-based convolutional neural network (faster R-CNN) was applied to contaminated real-world images taken from concrete bridges with complex backgrounds, including handwriting. A dataset of 5,009 cropped images was generated and labeled for two different objects, cracks and handwriting. The proposed network was then trained and tested using the generated image dataset. Four full-scale images that contained complex disturbance information were used to assess the performance of the trained network. The results of this study demonstrate that faster R-CNN can automatically locate crack from raw images, even with the presence of handwriting scripts. For comparative study, the proposed network is also compared with You Only Look Once v2 detection technique.