Adaptive PID-based deep reinforcement learning for load frequency control in islanded microgrids with heterogeneous resources and energy storage

Abstract

Load frequency control (LFC) is an essential measure in maintaining stability in power systems in islanded microgrids that include heterogeneous generation sources and energy storage systems. Island microgrid systems (IMGs) that integrate renewable energy sources widely use proportional-integral-derivative (PID) controllers for LFC. However, the overall control performance is highly sensitive to the accurate tuning of PID controller parameters. To address this problem, this paper proposes an adaptive PID tuning approach that studies the individual evaluations of the Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning algorithms. The proposed reinforcement learning (RL)-based adaptive PID tuning approach is used to adaptively regulate PID gains under the inherent uncertainties of the IMGs environment. The proposed RL-PID controller approach employs an agent that is trained offline through repeated interactions with the IMGs model, where a suitable reward function guides the learning process toward an optimal control policy. Once trained, the agent is implemented online to continuously update the PID gains for coordinated control of the Wind Turbine Generator (WTG), Solar Photovoltaic (SPV), Fuel Cell (FC) units, Electric Vehicle (EV), and Biogas Turbine Generator (BTG), ensuring effective load demand tracking. Simulations of an IMGs demonstrate that both DDPG-based PID controllers and TD3-based PID controllers outperform conventional PID controllers in dynamic response, settling time, and robustness to disturbances. Furthermore, the TD3-PID shows better stability and reduced oscillations compared to the DDPG-PID, which can be explained by the fact that it improves the policy update mechanism, leading to more effective adjustments in response to system changes. Although studied PID tuning approaches incorporate intelligent mechanisms for gain adjustment, the results indicate that the RL-based adaptive PID controller provides improved overall performance.