The power system inertia is gradually decreasing with the growing share of variable renewable energy (VRE). This may jeopardize the frequency dynamics and challenges the secure operation of power systems. In this paper, the concept of frequency security margin is proposed to quantify the system frequency regulation ability under contingency. It is defined as the maximum power imbalance that the system can tolerate while keeping frequency within the tolerable frequency range. A frequency constrained unit commitment (FCUC) model considering frequency security margin is proposed. Firstly, the analytical formulation of system frequency nadir is derived while considering both the frequency regulation characteristics of the thermal generators and the frequency support from VRE plants. Then, the frequency security margin is analytically formulated and piecewise linearized. A novel FCUC model is proposed by incorporating linear frequency security constraints into the traditional unit commitment model. Case studies on a modified IEEE RTS-79 system and HRP-38 system are provided to verify the effectiveness of the proposed FCUC model. The impacts of VRE penetration on system frequency security are analyzed using frequency security margin.