Search for: Search Button - Le CRAN
- Recherche
- Emplois-stages
- Formations
- Partenariats
- International

Ph. D. Project

Physics-informed data-driven system identification for data-efficient reinforcement learning for control

Dates:

2023/10/01 - 2026/09/30

Student:

Supervisor(s):

Description:

Modeling and simulation of systems are still of critical importance for characterizing the underlying dynamics of time-varying nonlinear system. A recent

solution has been to exploit emerging methods from machine learning and deep learning.

To facilitate the fundamental needs of system identification of time varying nonlinear dynamics, physics-informed neural networks (PINN) has emerged as a

hybrid deep learning paradigm that imbues machine learning models with physical domain knowledge such as physical laws and constraints emanating from

basic Newtonian physics-based laws (Raissi et al. 2019). Within this paradigm, neural networks are trained to solve supervised learning tasks while

respecting any given laws of physics described by general nonlinear differential equations (Stiasny et al. 2021).The distinguishing feature of PINN is the

synergistic integration of data-driven methods that reflect system dynamics in real-time with the governing physics.

Additionally, Koopman operator based approach for identification of nonlinear dynamics has garnered an extensive attention in the last 5 years (Bevanda et

al. 2021). The Koopman operator allows for handling nonlinear systems through a globally linear representation. In general, the operator is infinite-

dimensional - necessitating finite approximations - for which there is no overarching framework. Also, Koopman operator theory has long-standing

connections to known system-theoretic and dynamical system notions that are not universally recognized. Although there are principled ways of learning

such finite approximations, they are in many instances overlooked in favor of, often ill-posed and unstructured methods. When developed in conjunction

with Deep neural networks, the so called Deep Koopman operators have shown remarkable results in identifying nonlinear dynamics over finite linear

approximations (Lusch et al. 2018; Mauroy and Goncalves 2019).

However, so far, the potential of combining PINNs with Deep Koopman operators for identification of nonlinear dynamical systems has not been explored

in a detailed manner. It is believed that leveraging Koopman operator-based approach with PINNs shall result in high-fidelity and reduced models that are

well aligned with generally known physics properties as well as "interpretable".

On the other hand, recent advancements in the domain of adaptive dynamic programming and Reinforcement learning (ADP-RL) have led to remarkable

results in optimal control design for non-linear systems in the absence of system knowledge (complete or partial) (Kiumarsi et al. 2017). RL is a mature field

with well-established mathematical grounds for optimal (sub-optimal) control of non-linear dynamical systems in continuous as well as discrete time where

in the optimal control synthesis is largely based upon iterative solution for non-linear Hamilton-Jacobi-bellman equation (HJB) using neural network-based

structure (Mu et al. 2016). However, while the existing approaches provide exceptional asymptotic performance, trial-and-error based reinforcement

learning suffers from being data inefficient, i.e., requiring an excessive amount of trial data for training. This hampers the training quality as well as leads to

semi-global performance. Moreover, there is no insight/explanation about the system model learnt by the RL agents in an implicit manner. Similar is the

case with the nonlinear control policy learnt within the RL framework i.e., there is very little or no understanding available about the dynamics of the learnt

control policy. These aforementioned factors oblige training with excessive data and training duration (episodes) that are necessarily long.

Very recent efforts have been made by representing the environment dynamics as a linear dynamical system in a high-dimensional space (Han et al. 2020),

wherein the Koopman operator theory allows incorporating effective optimal control methods to produce high-quality trials thus accelerating learning (Shi

and Meng 2022).

While the first phase of the thesis will target development of PINN enabled Deep Koopman operator-based approaches for nonlinear system identification,

the second phase will investigate the appropriate integration and exploitation of such models within the deep reinforcement learning framework to render

the (sub)optimal control learning process data-efficient and interpretable. Overall, the goal will be to use recent data-driven identification methods and

Koopman theory to build adaptive, real-time models that can be integrated with reinforcement learning control frameworks. The field of application will be

robotics.

References

Bevanda, Petar, Stefan Sosnowski, and Sandra Hirche. 2021. "Koopman Operator Dynamical Models: Learning, Analysis and Control." Annual Reviews in

Control 52: 197-212.

Kiumarsi, Bahare, Kyriakos G Vamvoudakis, Hamidreza Modares, and Frank L Lewis. 2017. "Optimal and Autonomous Control Using Reinforcement

Learning: A Survey." IEEE Transactions on Neural Networks and Learning Systems 29 (6): 2042-62.

Lusch, Bethany, J Nathan Kutz, and Steven L Brunton. 2018. "Deep Learning for Universal Linear Embeddings of Nonlinear Dynamics." Nature

Communications 9 (1): 1-10.

Mu, Chaoxu, Zhen Ni, Changyin Sun, and Haibo He. 2016. "Data-Driven Tracking Control with Adaptive Dynamic Programming for a Class of

Continuous-Time Nonlinear Systems." IEEE Transactions on Cybernetics 47 (6): 1460-70.

Raissi, Maziar, Paris Perdikaris, and George E Karniadakis. 2019. "Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward

and Inverse Problems Involving Nonlinear Partial Differential Equations." Journal of Computational Physics 378: 686-707.

Shi, Haojie, and Max Q-H Meng. 2022. "Deep Koopman Operator with Control for Nonlinear Systems." IEEE Robotics and Automation Letters 7 (3):

7700-7707.

Stiasny, Jochen, George S Misyris, and Spyros Chatzivasileiadis. 2021. "Physics-Informed Neural Networks for Non-Linear System Identification for Power

System Dynamics." In 2021 IEEE Madrid PowerTech, 1-6.

solution has been to exploit emerging methods from machine learning and deep learning.

To facilitate the fundamental needs of system identification of time varying nonlinear dynamics, physics-informed neural networks (PINN) has emerged as a

hybrid deep learning paradigm that imbues machine learning models with physical domain knowledge such as physical laws and constraints emanating from

basic Newtonian physics-based laws (Raissi et al. 2019). Within this paradigm, neural networks are trained to solve supervised learning tasks while

respecting any given laws of physics described by general nonlinear differential equations (Stiasny et al. 2021).The distinguishing feature of PINN is the

synergistic integration of data-driven methods that reflect system dynamics in real-time with the governing physics.

Additionally, Koopman operator based approach for identification of nonlinear dynamics has garnered an extensive attention in the last 5 years (Bevanda et

al. 2021). The Koopman operator allows for handling nonlinear systems through a globally linear representation. In general, the operator is infinite-

dimensional - necessitating finite approximations - for which there is no overarching framework. Also, Koopman operator theory has long-standing

connections to known system-theoretic and dynamical system notions that are not universally recognized. Although there are principled ways of learning

such finite approximations, they are in many instances overlooked in favor of, often ill-posed and unstructured methods. When developed in conjunction

with Deep neural networks, the so called Deep Koopman operators have shown remarkable results in identifying nonlinear dynamics over finite linear

approximations (Lusch et al. 2018; Mauroy and Goncalves 2019).

However, so far, the potential of combining PINNs with Deep Koopman operators for identification of nonlinear dynamical systems has not been explored

in a detailed manner. It is believed that leveraging Koopman operator-based approach with PINNs shall result in high-fidelity and reduced models that are

well aligned with generally known physics properties as well as "interpretable".

On the other hand, recent advancements in the domain of adaptive dynamic programming and Reinforcement learning (ADP-RL) have led to remarkable

results in optimal control design for non-linear systems in the absence of system knowledge (complete or partial) (Kiumarsi et al. 2017). RL is a mature field

with well-established mathematical grounds for optimal (sub-optimal) control of non-linear dynamical systems in continuous as well as discrete time where

in the optimal control synthesis is largely based upon iterative solution for non-linear Hamilton-Jacobi-bellman equation (HJB) using neural network-based

structure (Mu et al. 2016). However, while the existing approaches provide exceptional asymptotic performance, trial-and-error based reinforcement

learning suffers from being data inefficient, i.e., requiring an excessive amount of trial data for training. This hampers the training quality as well as leads to

semi-global performance. Moreover, there is no insight/explanation about the system model learnt by the RL agents in an implicit manner. Similar is the

case with the nonlinear control policy learnt within the RL framework i.e., there is very little or no understanding available about the dynamics of the learnt

control policy. These aforementioned factors oblige training with excessive data and training duration (episodes) that are necessarily long.

Very recent efforts have been made by representing the environment dynamics as a linear dynamical system in a high-dimensional space (Han et al. 2020),

wherein the Koopman operator theory allows incorporating effective optimal control methods to produce high-quality trials thus accelerating learning (Shi

and Meng 2022).

While the first phase of the thesis will target development of PINN enabled Deep Koopman operator-based approaches for nonlinear system identification,

the second phase will investigate the appropriate integration and exploitation of such models within the deep reinforcement learning framework to render

the (sub)optimal control learning process data-efficient and interpretable. Overall, the goal will be to use recent data-driven identification methods and

Koopman theory to build adaptive, real-time models that can be integrated with reinforcement learning control frameworks. The field of application will be

robotics.

References

Bevanda, Petar, Stefan Sosnowski, and Sandra Hirche. 2021. "Koopman Operator Dynamical Models: Learning, Analysis and Control." Annual Reviews in

Control 52: 197-212.

Kiumarsi, Bahare, Kyriakos G Vamvoudakis, Hamidreza Modares, and Frank L Lewis. 2017. "Optimal and Autonomous Control Using Reinforcement

Learning: A Survey." IEEE Transactions on Neural Networks and Learning Systems 29 (6): 2042-62.

Lusch, Bethany, J Nathan Kutz, and Steven L Brunton. 2018. "Deep Learning for Universal Linear Embeddings of Nonlinear Dynamics." Nature

Communications 9 (1): 1-10.

Mu, Chaoxu, Zhen Ni, Changyin Sun, and Haibo He. 2016. "Data-Driven Tracking Control with Adaptive Dynamic Programming for a Class of

Continuous-Time Nonlinear Systems." IEEE Transactions on Cybernetics 47 (6): 1460-70.

Raissi, Maziar, Paris Perdikaris, and George E Karniadakis. 2019. "Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward

and Inverse Problems Involving Nonlinear Partial Differential Equations." Journal of Computational Physics 378: 686-707.

Shi, Haojie, and Max Q-H Meng. 2022. "Deep Koopman Operator with Control for Nonlinear Systems." IEEE Robotics and Automation Letters 7 (3):

7700-7707.

Stiasny, Jochen, George S Misyris, and Spyros Chatzivasileiadis. 2021. "Physics-Informed Neural Networks for Non-Linear System Identification for Power

System Dynamics." In 2021 IEEE Madrid PowerTech, 1-6.

Keywords:

System identification, physics-informed neural networks, deep learning, reinforcement learning

Department(s):

Control Identification Diagnosis |