The productivity of construction equipment plays an important role in completing construction projects within schedule and under budget. However, the current productivity monitoring on construction sites highly depends on manually observing and recording equipment activities, which is labor-intensive and time-consuming. To address this problem, an increasing number of research studies focused on automatically identifying equipment activities from site surveillance videos . However, these studies failed to accurately conduct the activity recognition and productivity analysis, when multiple pieces of equipment are working together. This research proposes a novel framework for automatically analyzing the activity and productivity of multiple excavators . In this framework, three convolutional neural networks are designed to detect, track and recognize the activities of excavators. The results are further compiled to analyze excavator's activity time, working cycle, and productivity. The proposed framework has been tested with the videos recorded from real construction sites. The overall activity recognition has achieved 87.6% accuracy. The productivity calculation has achieved 83% accuracy, which indicates the feasibility of the proposed framework for automating the monitoring of excavator's productivity.