Context-aware Reinforcement Learning for RCS Design Space Exploration

Abstract

Reconfigurable Control Systems (RCS) operate in dynamic environments where increasing demands for reconfiguration introduce significant uncertainties that complicate the design process. A major challenge is designing RCS with a Configuration Space (CS) that adapts to evolving operational contexts while maintaining fundamental system properties such as safety. In this paper, we propose Reconfiguration Control Q-Learning (ReCQ-Learning), an enhanced Q-learning algorithm for context-aware CS exploration. Our approach implements the reconfiguration controller as a Reinforcement Learning Reconfiguration Agent (RLRA) that recognizes and adapts to different reconfiguration contexts: Predefined-to-Predefined (P2P), Predefined-to-Foreseeable (P2F), and Predefined-to-New (P2N). These contexts represent increasing levels of uncertainty in the reconfiguration environment. The algorithm introduces novel exploration strategies that leverage this context taxonomy alongside validity constraints to efficiently guide the RLRA through the configuration space. This exploration algorithm serves dual purposes: during the offline design phase, it enables the RLRA to generate a valid initial CS by exploring primarily P2P and P2F contexts; during runtime, it handles reconfiguration triggers across all context types by learning optimized policies, with particular emphasis on developing robust strategies for P2N scenarios. The runtime learning feedback can then be used to improve the initial design specifications, creating an iterative improvement cycle between design and operation phases. Experimental validation using the FESTO CP Factory case study demonstrates that ReCQ-Learning significantly outperforms traditional approaches in exploration efficiency, configuration validity, and context adaptation across all reconfiguration scenarios.