Journal of Applied Research in Electrical Engineering

Journal of Applied Research in Electrical Engineering

DDPG-Based PID Gain Scheduling for DC Motor Speed Control: Exponential Reward and Heuristic Action Bounds

Document Type : Research article

Authors
1 Department of Mechanical Engineering, Islamic Azad University, Saveh, Iran
2 Department of Computer Engineering, Islamic Azad University, Qazvin, Iran
Abstract
This paper presents a deep reinforcement learning approach for static gain scheduling of a PID controller in a DC motor speed control system. A deep deterministic policy gradient (DDPG) agent learns a direct mapping from the speed reference to fixed PID gains that remain constant during each ten second episode. The action space is constrained by heuristic bounds derived from domain knowledge, ensuring that the proportional, integral, and derivative gains stay within safe ranges. An exponential reward function, which heavily penalises the integrated squared tracking error, is used to force the agent to virtually eliminate the steady state offset. The DC motor model yields a second order overdamped transfer function. After training over 1000 episodes with random references between 0.5 and 2.0 rad/s, the learned gains for a reference of 1.0 rad/s are compared with those obtained from MATLAB’s PID Tuner. The reinforcement learning scheduler achieves approximately 34 % faster rise time, 23 % lower IAE, and 27 % lower ITAE than the classical tuner, although with a larger overshoot (13.1 % versus 7.04 %). Robustness analysis under ±20 % parameter variations shows that the RL controller maintains lower IAE in most scenarios. The training was repeated over five independent runs, showing very low variance (IAE standard deviation: 0.0019 rad·s), which confirms reproducibility. The main limitation is the static policy, where gains are not updated within an episode. The proposed method offers a competitive alternative to classical tuning when high tracking accuracy is required and the operating point changes slowly.

Highlights

  • Exponential reward function for DDPG‑based PID gain scheduling that heavily penalizes integrated squared error, virtually eliminating steady‑state offset.
  • Heuristic action bounds ensuring safe exploration and deployment of PID gains (Kp∈ [5,12], Ki∈[20,50], Kd∈[0.2,0.6]) without destabilizing the plant.
  • Static gain scheduling mapping speed reference to fixed PID gains per episode, simplifying deployment and avoiding high‑frequency gain adaptation.
  • Rigorous reproducibility analysis with five independent training runs, showing IAE standard deviation as low as 0.0019 rad·s.
  • Systematic robustness evaluation under ±20% parameter variations and hyperparameter sensitivity analysis, confirming RL controller performance.

Keywords
Subjects

[1]    P. P. Lu, W. Huang, and J. Xiao, "Speed tracking of brushless DC motor based on deep reinforcement learning and PID," in Proc. 2021 7th Int. Conf. on Condition Monitoring of Machinery in Non-Stationary Operations (CMMNO), Jun. 2021, pp. 130–134.  DOI: 10.1109/CMMNO53328.2021.9467649
[2]    T. Shuprajhaa, S. K. Sujit, and K. Srinivasan, "Reinforcement learning based adaptive PID controller design for control of linear/nonlinear unstable processes," Appl. Soft Comput., vol. 128, p. 109450, 2022. DOI: 10.1016/j.asoc.2022.109450
[3]    E. Okafor, D. Udekwe, Y. Ibrahim, M. Bashir Mu’azu, and E. G. Okafor, "Heuristic and deep reinforcement learning-based PID control of trajectory tracking in a ball-and-plate system," J. Inf. Telecommun., vol. 5, no. 2, pp. 179–196, 2021. DOI: 10.1080/24751839.2020.1833137
[4]    M. H. K. Fadaei, S. G. R. A. G. Atigh, A. Zalaghi, and Z. Torkani, "Design of model reference adaptive control for a 4-dof serial manipulator," in 2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI), 2019, pp. 108–112. DOI: 10.1109/KBEI.2019.8735078
[5]    A. M. Hassan, J. Ababneh, H. Attar, T. Shamseldin, A. Abdelbaset, and M. E. Metwally, "Reinforcement learning algorithm for improving speed response of a five-phase permanent magnet synchronous motor based model predictive control," PLoS ONE, vol. 20, no. 1, p. e0316326, 2025. DOI: 10.1371/journal.pone.0316326
[6]    K. Sinaei and M. R. S. Yazdi, "PID controller tuning with deep reinforcement learning policy gradient methods," in Proc. 29th Int. Conf. Iranian Soc. Mech. Eng. & 8th Conf. Thermal Power Plants, Tehran, Iran, May 2021, pp. 25–27.
[7]    S. M. Al Selwi et al., "Deep deterministic policy gradient algorithm: A systematic review," Heliyon, vol. 10, no. 9, e30697, 2024. DOI: 10.1016/j.heliyon.2024.e30697
[8]    X. Hao, Z. Xin, W. Huang, S. Wan, G. Qiu, T. Wang, and Z. Wang, "Deep reinforcement learning enhanced PID control for hydraulic servo systems in injection molding machines," Sci. Rep., vol. 15, no. 1, p. 23005, 2025. DOI: 10.1038/s41598-025-05904-2
[9]    R. Shahouni, M. Bahraini, M. Abrofarakh, and M. Abbasi, "Adaptive tuning of fractional order PID controllers for nonlinear processes using hybrid PSO DQN reinforcement learning," Sci. Rep., vol. 15, no. 1, p. 38545, 2025. DOI: 10.1038/s41598-025-22509-x
[10]  M. H. K. Fadaei, S. A. H. Pahnehkolaei, and M. J. Hesarlou, "Design of hybrid fuzzy-PD controller for an inverted pendulum," Majlesi J. Mechatronic Syst., vol. 10, no. 2, pp. 21–27, 2021. DOI: 10.52547/mjms.10.2.21
[11]  H. Sarma and A. Bardalai, "An intelligent PID controller tuning for speed control of BLDC motor using driving training-based optimization," Int. J. Power Electron. Drive Syst. (IJPEDS), vol. 14, no. 4, pp. 2474–2486, 2023. DOI: 10.11591/ijpeds.v14.i4.pp2474-2486
[12]  N. Ajlouni, A. Osyavas, F. Ajlouni, A. Almassri, F. Takaoglu, and M. Takaoglu, "Enhancing PID control robustness in CSTRs: a hybrid approach to tuning under external disturbances with GA, PSO, and machine learning," Neural Comput. Appl., pp. 12153–12177, 2025. DOI: 10.1007/s00521-025-11170-0
[13]  N. Rahoua, H. Benguesmia, R. Mechgoug, and N. Tkouti, "Comparative performance analysis of PID and fuzzy-PID speed controllers for brushless DC motor drives," Eng., Technol. Appl. Sci. Res., vol. 15, no. 5, pp. 27985–27992, 2025. DOI: 10.48084/etasr.13532
[14]  P. Gaur, V. P. Singh, T. Varshney, P. Sanjeevikumar, and A. Mathur, "Design of twin delayed deep deterministic policy gradient RL based adaptive controller for DC motor speed regulation considering uncertainties," Sci. Rep., vol. 16, no. 1, p. 3622, 2026. DOI: 10.1038/s41598-025-33644-w
[15]  B. Ke, T. Qiu, and Y. Shen, "Deep reinforcement learning in continuous control: Advances and challenges," ITM Web of Conferences, vol. 78, 01021, 2025. DOI: 10.1051/itmconf/20257801021
[16]  V. Nazarzehi Had and Z. Raisi, “Deep Learning based Underwater Object Detection and Recognition for Multi-Robot Systems,” Journal of Applied Research in Electrical Engineering, vol. 4, no. 1, pp. 24–33, Jan. 2025. DOI: 10.22055/jaree.2025.48211.1142
[17]  M. L. Richter, L. Malihi, A. K. P. Windler, and U. Krumnack, “Analyzing the Inference Process in Deep Convolutional Neural Networks using Principal Eigenfeatures, Saturation and Logistic Regression Probes,” Journal of Applied Research in Electrical Engineering, vol. 2, no. 1, pp. 1–10, Jan. 2023. DOI: 10.22055/jaree.2022.40073.1049
[18]  S. Ahmed, U. Tariq, A. Hasan and H. Rehman, "A Deep Reinforcement Learning Paradigm for DC Motor Speed Control," 2025 IEEE International Electric Machines & Drives Conference (IEMDC), Houston, TX, USA, 2025, pp. 709-714, DOI: 10.1109/IEMDC60492.2025.11060983
[19]  S. Tufenki, G. Kavuran, and C. Yeroglu, "An approach for DC motor speed control with off policy reinforcement learning method," Balkan J. Electr. Comput. Eng., vol. 11, no. 2, pp. 184–189, 2023. DOI: 10.17694/bajece.1114868
[20]  S. Tufenkci, B.  Alagoz, G. Kavuran, C. Yeroglu, N. Herencsar & S. Mahata, "Implementations of TD3 and DDPG reinforcement learning techniques for tuning PID controller of TRMS system," Journal of Systems Science and Systems Engineering, 2025. DOI: https://doi.org/10.1007/s11518-025-5693-5
[21]  A. M. Davatgar, H. Mojallali, and X. Li, "Designing an optimal PID controller for a gas turbine system using reinforcement learning," International Transactions on Electrical Energy Systems, vol. 2025, Article 1376194, 2025. DOI: 10.1155/etep/1376194
[22]  U. Demircioglu and H. Bakır, "Reinforcement learning–driven proportional–integral–derivative controller tuning for mass–spring systems," Engineering Applications of Artificial Intelligence., vol. 162, 112692, 2025. DOI: 10.1155/etep/1376194
[23]  U. Kruthika and S. Paneerselvam, "Novel multiagent reinforcement learning framework using twin delayed deep deterministic policy gradient for adaptive PID control in boiler turbine systems," Sci. Rep., vol. 15, 34558, 2025. DOI: 10.1038/s41598-025-17928-9
[24]  S. M. Ghamari et al., "Design of a robust adaptive cascade fractional order PID controller enhanced by RL algorithm for speed regulation of BLDC motor in EVs," Energies, vol. 18, no. 19, 5056, 2025. DOI: https://doi.org/10.3390/en18195056
[25]  K. Vijaya Lakshmi and M. Manimozhi, "Real time HIL implementation of DDPG based reinforcement learning controller for a DC servo motor," IEEE Access, vol. 13, pp. 1–15, 2025. DOI: 10.1109/ACCESS.2025.3638051
[26]  A. Zribi and H. Taouil, "Adaptive reinforcement learning based tuning of neural PID controllers for nonlinear systems," J. Vib. Control, 2025. DOI: https://doi.org/10.1177/10775463251404852
[27]  N. Maleki and E. Moradi, “Robust H∞ Control of DC Motor in the Presence of Input Delay and Disturbance by the Predictor‐Based Method,” Complexity, vol. 2022, no. 1, p. 1902166, 2022. DOI: 10.1155/2022/1902166
[28]  N. Maleki and E. Moradi, “Designing a Robust Memory State Feedback Controller Leveraging LMI on DC Motor Under Time Delays and Norm-Bounded Disturbances,” Modeling and Simulation in Electrical and Electronics Engineering, vol. 4, no. 2, pp. 43–55, 2024. DOI: 10.22075/mseee.2025.36095.1191
[29]  M. A. L. Khaniki and E. Moradi, “Delay‐Scheduled Adaptive Observer Control Strategy for Nonlinear Systems With Time‐Varying Delays,” IET Control Theory & Applications, vol. 20, no. 1, p. e70126, 2026. DOI: 10.1049/cth2.70126
Volume 5, Issue 1
January 2026
Pages 12-24

  • Receive Date 22 May 2026
  • Revise Date 19 June 2026
  • Accept Date 10 August 2026