IEEE Communications Surveys & Tutorials

Sixth-generation (6G) cellular systems will have an inherent vulnerability to physical (PHY)-layer attacks and privacy leakage, due to the large-scale heterogeneous networks with booming time-sensitive applications. Important wireless techniques including non-orthogonal multiple access, mobile edge computing, millimeter-wave, massive multiple-input and multiple-output, visible light communication, terahertz, and intelligent reflecting surface can improve the spectrum efficiency and quality-of-service but will raise challenges for the 6G PHY and cross-layer security and privacy protection. Existing optimization based PHY and cross-layer security and privacy protection schemes such as the convex optimization method have to rely on accurate attack patterns and strategies and thus suffer from performance degradation in 6G systems that have shorter communication latency, more devices and higher spectrum efficiency than 5G. Reinforcement learning (RL) algorithms help wireless devices optimize their security policies to enhance the security performance in dynamic networks against smart attacks without depending on the attack model. Therefore, this article provides a comprehensive survey on the RL based 6G PHY cross-layer security and privacy protection. In this article, we investigate the potential attacks in 6G systems and discuss the PHY cross-layer security solutions. A brief overview of reinforcement learning algorithms is provided. Afterward, we review the RL based PHY-layer security and privacy protection and discuss how to apply RL algorithms in 6G security scenarios, especially focusing on the game with jammers, eavesdroppers, spoofers and inference attackers. The RL based security solutions for unmanned aerial vehicles (UAVs) and cross-layer scenarios are also reviewed. The future research directions are identified and the corresponding RL based potential solutions are discussed for 6G.

Reinforcement Learning-Based Physical Cross-Layer Security and Privacy in 6G