Ph. D. Project
Title:
DEEP REINFORCEMENT LEARNING AND OPTIMIZATION FOR STATE ESTIMATION AND CONTROL DESIGN OF INTERCONNECTED
NONLINEAR SYSTEMS UNDER CONSTRAINTS
Dates:
2025/10/01 - 2028/09/30
Supervisor(s):
Other supervisor(s):
OUDANI Mustapha (mustapha.oudani@uir.ac.ma)
Description:
This doctoral thesis project is part of the scientific collaboration between CRAN - University of Lorraine and TIC Lab - International
University of Rabat. It aims to develop a methodological approach based on AI techniques to address issues related to state estimation
(or software sensors) and control of interconnected nonlinear systems under constraints, for which model-based approaches have not
provided a solution to date.
A broad class of dynamic systems, whether physical or artificial, can be modeled by interconnected nonlinear differential equations:
Communication, transportation (Oudani et al. 2024) or energy networks (Thabet et al. 2010, Jadbabaie et al., 2020; Ren & Beard, 2020).
These interconnected nonlinear systems often operate under significant constraints such as limited measurement capabilities, restricted
communication bandwidth and/or limitations in terms of energy or computation, which negatively affect their observability and
controllability performances (Mu et al., 2023; Li et al., 2023). These limitations are particularly
problematic in decentralized and distributed systems, where state estimation must be performed with limited information exchange, while
the presence of unknown inputs, external disturbances, and model uncertainties further complicates the design of robust observers and
controllers (Boutat-Baddas et al., 2021; Bel Haj Frej et al., 2016, Kang et al., 2023; Narayanan & Jagannathan, 2017; Nejabat & Nikoofard,
2023).
The use of hybrid approaches based on both models and Artificial Intelligence techniques opens up a very promising field but remains
relatively unexplored. In a recent study (Yaghmaie et al. 2019) based on reinforcement learning technique, with specific activation
functions, the authors show that there could be solutions to optimization problems under constraints for a class of linear dynamic
systems, where the model based approach that has not provided a solution to date.
Other recent research has emphasized the integration of machine learning techniques to enhance predictive accuracy, optimize decisionmaking
under constraints, and develop AI driven control strategies capable of handling uncertainty and incomplete data scenarios (Liu et
al., 2023). Particularly, reduced-order modeling techniques significantly reduce computational
complexity, while reinforcement learning-based control strategies provide adaptive and decentralized decision-making mechanisms.
It is clear that approaches based solely on models for estimating and controlling interconnected nonlinear systems under constraints
quickly show their limitations and impose very restrictive conditions to ensure a minimum level of performance.
Expected research should focus on developing more sophisticated learning frameworks, particularly through the integration of deep
reinforcement learning.
The first part of the thesis will focus on the development of a distributed approach to build deep neural networks for each subsystem of
the interconnected model while minimizing a global criterion that considers all interactions in the network. The reward function would
consider the global dynamic model and will be used to ensure minimum performance. The
construction of these networks (universal approximators) will be inspired by a recent work (Farkane et al. 2024) that we have developed to
solve a class of nonlinear PDEs (large scale systems, for which we obtained remarkable performance in terms of accuracy and speed
compared to Eulerian discretization techniques).
Construction of distributed strategies for reinforcement learning based state estimation and control design will focus on cases where:
⬢ Sensors of each subsystem are limited
⬢ Communication is restricted
⬢ Energy is limited
The second part of the thesis, equally important, will focus on evaluating these AI-based algorithms in terms of robustness, accuracy, and
computation time through numerical simulation of various types of interconnected networks under constraints.
University of Rabat. It aims to develop a methodological approach based on AI techniques to address issues related to state estimation
(or software sensors) and control of interconnected nonlinear systems under constraints, for which model-based approaches have not
provided a solution to date.
A broad class of dynamic systems, whether physical or artificial, can be modeled by interconnected nonlinear differential equations:
Communication, transportation (Oudani et al. 2024) or energy networks (Thabet et al. 2010, Jadbabaie et al., 2020; Ren & Beard, 2020).
These interconnected nonlinear systems often operate under significant constraints such as limited measurement capabilities, restricted
communication bandwidth and/or limitations in terms of energy or computation, which negatively affect their observability and
controllability performances (Mu et al., 2023; Li et al., 2023). These limitations are particularly
problematic in decentralized and distributed systems, where state estimation must be performed with limited information exchange, while
the presence of unknown inputs, external disturbances, and model uncertainties further complicates the design of robust observers and
controllers (Boutat-Baddas et al., 2021; Bel Haj Frej et al., 2016, Kang et al., 2023; Narayanan & Jagannathan, 2017; Nejabat & Nikoofard,
2023).
The use of hybrid approaches based on both models and Artificial Intelligence techniques opens up a very promising field but remains
relatively unexplored. In a recent study (Yaghmaie et al. 2019) based on reinforcement learning technique, with specific activation
functions, the authors show that there could be solutions to optimization problems under constraints for a class of linear dynamic
systems, where the model based approach that has not provided a solution to date.
Other recent research has emphasized the integration of machine learning techniques to enhance predictive accuracy, optimize decisionmaking
under constraints, and develop AI driven control strategies capable of handling uncertainty and incomplete data scenarios (Liu et
al., 2023). Particularly, reduced-order modeling techniques significantly reduce computational
complexity, while reinforcement learning-based control strategies provide adaptive and decentralized decision-making mechanisms.
It is clear that approaches based solely on models for estimating and controlling interconnected nonlinear systems under constraints
quickly show their limitations and impose very restrictive conditions to ensure a minimum level of performance.
Expected research should focus on developing more sophisticated learning frameworks, particularly through the integration of deep
reinforcement learning.
The first part of the thesis will focus on the development of a distributed approach to build deep neural networks for each subsystem of
the interconnected model while minimizing a global criterion that considers all interactions in the network. The reward function would
consider the global dynamic model and will be used to ensure minimum performance. The
construction of these networks (universal approximators) will be inspired by a recent work (Farkane et al. 2024) that we have developed to
solve a class of nonlinear PDEs (large scale systems, for which we obtained remarkable performance in terms of accuracy and speed
compared to Eulerian discretization techniques).
Construction of distributed strategies for reinforcement learning based state estimation and control design will focus on cases where:
⬢ Sensors of each subsystem are limited
⬢ Communication is restricted
⬢ Energy is limited
The second part of the thesis, equally important, will focus on evaluating these AI-based algorithms in terms of robustness, accuracy, and
computation time through numerical simulation of various types of interconnected networks under constraints.
Department(s):
| Control Identification Diagnosis |