Forward Error Correction (FEC) is critical for packet loss recovery in today’s real-time video conferencing applications to improve video quality. However, it is a challenging task to determine an appropriate FEC redundancy rate under dynamic network conditions. Therefore, we propose RL-AFEC, which is a Reinforcement Learning (RL)-based FEC scheme that can adaptively change redundancy rates to effectively improve video quality while introducing low additional bandwidth consumptions.
This work is an academic project collaboration with Fortinet, Inc., CA, USA, under the supervision of Prof. H. Jonathan Chao at the NYU High-speed Networking Lab. As the project supervisor, I led several talented MS students in our research group to work together on this project and completed this work.
Project Overview:
(1) Scenario: Implement an RL-based Adaptive FEC (AFEC) scheme for real-time video streams at the SD-WAN gateways. The objective is to intelligently select a proper redundancy rate for each video frame in dynamic network conditions to improve the reconstructed video quality with low bandwidth consumption
(2) Generate a video dataset by adding different levels of impairment (packet loss) to the original video to get the degraded videos, where the average packet loss rates are controlled by the Gilbert-Elliot (GE) channel, which is a widely-used two-state Markov model for simulating random losses
(3) Propose a Packet-Level Video Quality Assessment (PL-VQA) method with Supervised Learning (SL) to accurately estimate the received video quality in real time without the need to reconstruct visual contents
(4) RL design for AFEC: Chooses a redundancy rate for each of the first 𝐾 critical frames individually to protect critical video contents, and then selects a redundancy rate that will be applied to all the remaining non-critical frames
(5) Evaluation results: RL-AFEC can achieve good video quality for more than 95% of 1-second videos with only 40% additional bandwidth consumption
(6) RL-AFEC can efficiently avoid video distortions under dynamic packet losses with low bandwidth usage for sending redundant FEC packets
Contributions:
- We designed a no-reference packet-level video quality assessment method based on Video Multimethod Assessment Fusion (VMAF) and applied Supervised Learning (SL) to generate real-time video quality scores without the need of reconstructing visual contents.
- We exploited RL to determine appropriate redundancy rates in fluctuating network conditions to mitigate the impact of packet loss with an aim to mitigate bandwidth consumption, and then added redundant packets based on the frame-level Reed-Solomon (RS) code.
- The proposed solution achieved good video quality in >95% of cases with only 40% additional bandwidth consumption for redundant packets, which outperforms all baseline methods.
Abstract:
Real-time video communication is profoundly changing people’s lives, especially in today’s pandemic situation. However, packet loss during video transmission degrades reconstructed video quality, thus impairing users’ Quality of Experience (QoE). Forward Error Correction (FEC) techniques are commonly employed in today’s audio and video conferencing applications, such as Skype and Zoom, to mitigate the impact of packet loss. FEC helps recover the lost packets during transmissions at the receiver side, but the additional bandwidth consumption is also a concern. Since network conditions are highly dynamic, it is not trivial for FEC to maintain video quality with a fixed bandwidth overhead.
In this paper, we propose RL-AFEC, an adaptive FEC scheme based on Reinforcement Learning (RL) to improve reconstructed video quality with an aim to mitigate bandwidth consumption for different network conditions. RL-AFEC learns to select a proper redundancy rate for each video frame, and then adds redundant packets based on the frame-level Reed-Solomon (RS) code. We also implement a novel packet-level Video Quality Assessment (VQA) method based on Video Multimethod Assessment Fusion (VMAF), which leverages Supervised Learning (SL) to generate video quality scores in real time by only extracting information from the packet stream without the need of visual contents. Extensive evaluations demonstrate the superiority of our scheme over other baseline FEC methods.
Publications:
- [MMSys ’22] Ke Chen, Han Wang, Shuwen Fang, Xiaotian Li, Minghao Ye, and H. Jonathan Chao, “RL-AFEC: Adaptive Forward Error Correction for Real-time Video Communication Based on Reinforcement Learning,” The 13th ACM Multimedia Systems Conference (MMSys), 2022. (Acceptance rate: 34%, 16/46) [Paper URL] [Codes] [PDF]