Drones in Internet of Drones (IoD) can be able to reconnoiter environment, transport the commodity with the help of embedded various sensors. They have been widely used in various fields and brought a great convenience to the production and life. But data collected by sensors embedded in drones are facing new security challenges and privacy issues with the technology update over time. For the sake of ensuring the security of transmitted data, many authentication and key agreement (AKA) schemes have been proposed in the past. Nevertheless, most of schemes are subjected to serious security risks and have high communication and computation cost. To address these issues in IoD, we propose a lightweight AKA scheme in which there are only secure one-way hash function and bitewise XOR operations when drones and users mutually authenticate each other. The proposed scheme can achieve AKA-security under the random oracle model and withstand various known attacks. Meanwhile, the security comparison demonstrates our proposed scheme provides better security. In terms of communication and computation cost, our proposed scheme has better functionality features than the other two schemes.