Multimodal Deep Reinforcement Learning for Visual Security of Virtual Reality Applications

Abstract

The rapid development of virtual reality (VR) technologies is bringing unprecedented immersive experiences and unusual digital content. Nevertheless, these advancements introduce new security challenges, especially in safeguarding the visual content displayed by VR devices like VR glasses and head-mounted displays. Most existing approaches for visual output security rely exclusively on numerical data, such as object attributes and overlook the need of visual information necessary for thorough VR protection. Moreover, these approaches typically assume a fixed size input, failing to address the dynamic nature of VR where the number of virtual items is constantly changing. This article presents a multimodal deep reinforcement learning (MMDRL) approach to secure the visual outputs in VR applications. We formalize a Markov decision process (MDP) framework for the MMDRL agent that integrates both numerical and image data into the state space to effectively mitigate visual threats. Furthermore, our MMDRL agent is engineered to handle data of varying sizes, which makes it more suitable for VR environments. Results from our experiments demonstrate the agent’s ability to successfully counteract visual attacks, significantly outperforming previous approaches. The ablation study confirms the important role of image data in improving the agent’s performance, highlighting the efficacy of our multimodal approach. In addition, we provide a video demonstration to showcase these results. Finally, we open-source our VR testbed and source code for further testing and benchmarking.