Advanced Reinforcement Learning-Based Optimization Techniques for Wireless Access Networks

Abstract

Next-generation wireless networks have grown increasingly complex over the years due to the continuous demand generated by newer services like Extended Reality (XR), encompassing Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). 5th generation networks (5G) introduced three main network services to provide support to new traffic types with diverse quality of service demands: Enhanced Mobile Broadband (eMBB), Ultra-Reliable and Low Latency Communications (URLLC), and massive Machine Type Communications (mMTC). Despite the advancements achieved by 5G, there is still much room for improvement when meeting the needs of more data-intensive, low-latency, and ultra-high-reliability applications. In new generation Wi-Fi networks, Wi-Fi 6, 7, and the futuristic Wi-Fi 8, the race has been dedicated to providing extremely high throughput required by throughput-hungry services such as XR. Despite the success of the changes introduced in such amendments, Wi-Fi still falls short of providing ultra-high throughput with ultra-low latency and high reliability for applications such as cooperative mobile robots and others [1]. To suffice such complexity and myriad of requirements, Machine Learning (ML)-based solutions have stepped up and provided elegant and efficient solutions for many challenging wireless communications problems [2]. Several advances in ML in the field of computer vision, natural language processing, game playing, and robotics have allowed the migration of many of these applications to the field of wireless communications. More specifically, we foresee the immense application potential that Reinforcement Learning (RL), a subfield of ML, can provide due to its capacity to learn, as humans do, complex systems by interacting with an unknown environment. In this work, we aim to apply advanced state-of-the-art RL techniques on wireless access networks to optimize network performance. As case studies, we selected load balancing, handover in multi-RAT networks, XR codec selection, resource management in 4th, 5th, and beyond networks. We propose centralized and hierarchical RL-based solutions and explore the advantages of team learning and multi-agent reinforcement learning (MARL) in the proposed use cases. Our novel MARL algorithms demonstrate their capabilities to overperform centralized ones when the problem can be modeled as cooperative or competitive. Similarly, we study advanced state-of-the-art RL techniques in Wi-Fi networks. In this case, we explored spatial reuse, traffic allocation, and channel selection for IEEE 802.11ax and IEEE 802.11be networks. The results of this investigation underscore the effectiveness of various reinforcement learning (RL) techniques, especially in the multi-agent setting, across diverse application domains and wireless access networks. Firstly, competitive multi-agent reinforcement learning (MARL) schemes yield better results than centralized ones, enabling agents to compete more efficiently for resources in a load-balancing scenario. Secondly, hierarchical algorithms offer more optimal solutions in dual connectivity handovers compared to centralized ones, thanks to their unique capacity to handle sparse rewards. Thirdly, in Wi-Fi networks, cooperative spatial reuse appears to enhance collaboration among agents, and transfer reinforcement learning (TRL) facilitates quick adaptation in dynamic environments. Fourthly, additional methods need to be incorporated to address partially observable Markov decision process (POMDP) problems, a common characteristic in wireless networks, to enable successful utilization of RL. Finally, parallel transfer reinforcement learning (PTRL) can significantly improve convergence speed without the need for the classical teacher-student paradigm in sequential transfer reinforcement learning (TRL).