Ph. D. Project
Title:
Control by dynamic programming: robust stability guarantees
Dates:
2024/10/01 - 2027/09/30
Supervisor(s):
Other supervisor(s):
ANDRIEU Vincent
Description:
The objective of this PhD thesis is to contribute to the development of methodological tools for the synthesis
of (near-)optimal, stabilizing, and robust control laws for nonlinear dynamical systems. To this end, we will
focus on dynamic programming techniques. Dynamic programming is the preferred approach for building
efficient (near optimal) controllers for general system dynamics and cost functions. However, controllers
obtained through dynamic programming do not a priori come with stability and robustness guarantees, which
are essential in most control applications. The goal of this thesis is to identify conditions or even revisit
dynamic programming algorithms if needed to provide both performance and robust stability guarantees.
Various results have recently proposed conditions for endowing dynamic programming algorithms with
stabilizing properties, see, e.g., [1-5]. However, the fundamental question of the robustness to exogenous
disturbances and measurement noise is almost untouched, and only very recently first results for the linear
quadratic problem have appeared [6]. In our recent works in [7,8], we provide conditions under which stability
guarantees can be ensured for a general class of systems and general cost functions when the inputs are
generated by value iteration or policy iteration. We also show that the weakest form of robustness can be
guaranteed under mild conditions in this case, in the sense that arbitrarily small (vanishing) perturbations are
proved not to destroy the ensured stability property for the nominal system [9]. Still, the properties
established in [7,8] are not enough for most real-world applications and we need to go further by ensuring
stronger stability properties (input-to-state stability and Lp stability) for general nonlinear systems and cost
functions. This will allow quantifying the closed-loop system robustness, thereby clarifying the link between
the choice of the cost function and the system robustness properties and will open the door to the study of
interconnected systems.
References
[1] M. Ha, D. Wang, D. Liu (2021). Generalized value iteration for discounted optimal control with stability
analysis. Systems & Control Letters, 147, 104847.
[2] A. Heydari (2017). Stability analysis of optimal adaptive control under value iteration using a stabilizing
initial policy. IEEE Transactions on Neural Networks and Learning Systems, 29(9), 4522-4527.
[3] M. Granzotto, R. Postoyan, L. Buşoniu, D. Neić, J. Daafouz, J. (2020). Finite-horizon discounted optimal
control: stability and performance. IEEE Transactions on Automatic Control, 66(2), 550-565.
[4] A. Al-Tamimi, F.L. Lewis, M. Abu-Khalaf (2008). Discrete-time nonlinear HJB solution using approximate
dynamic programming: Convergence proof. IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics), 38(4), 943-949.
[5] Y. Jiang, Z.-P. Jiang (2017). Robust Adaptive Dynamic Programming. John Wiley & Sons.
[6] B. Pang, T. Bian, Z.-P. Jiang (2022). Robust policy iteration for continuous-time linear quadratic regulation.
IEEE Trans. on Automatic Control, 67(1), 504-511.
[7] M. Granzotto, O.L. De Silva, R. Postoyan, D. Nesic, Z.-P. Jiang (2022). Policy iteration: for want of recursive
feasibility, all is not lost. arXiv preprint arXiv:2210.14459.
[8] M. Granzotto, R. Postoyan, L. Buşoniu, D. Neić, J. Daafouz. (2020). Finite-horizon discounted optimal
control: stability and performance. IEEE Transactions on Automatic Control, 66(2), 550-565.
[9] C.M. Kellett, A.R. Teel (2004). Smooth Lyapunov functions and robustness of stability for difference
inclusions. Systems & Control Letters, 52(5), 395-405.
[10] S. Zoboli, V. Andrieu, D. Astolfi, G. Casadei, J.S. Dibangoye, M. Nadri. (2021, December). Reinforcement
learning policies with local LQR guarantees for nonlinear discrete-time systems. In 2021 60th IEEE Conference
on Decision and Control (CDC) (pp. 2258-2263).
[11] S. Zoboli, D. Astolfi, V. Andrieu, V. (2023). Total stability of equilibria motivates integral action in discrete-
time nonlinear systems. Automatica, 155, 111154.
of (near-)optimal, stabilizing, and robust control laws for nonlinear dynamical systems. To this end, we will
focus on dynamic programming techniques. Dynamic programming is the preferred approach for building
efficient (near optimal) controllers for general system dynamics and cost functions. However, controllers
obtained through dynamic programming do not a priori come with stability and robustness guarantees, which
are essential in most control applications. The goal of this thesis is to identify conditions or even revisit
dynamic programming algorithms if needed to provide both performance and robust stability guarantees.
Various results have recently proposed conditions for endowing dynamic programming algorithms with
stabilizing properties, see, e.g., [1-5]. However, the fundamental question of the robustness to exogenous
disturbances and measurement noise is almost untouched, and only very recently first results for the linear
quadratic problem have appeared [6]. In our recent works in [7,8], we provide conditions under which stability
guarantees can be ensured for a general class of systems and general cost functions when the inputs are
generated by value iteration or policy iteration. We also show that the weakest form of robustness can be
guaranteed under mild conditions in this case, in the sense that arbitrarily small (vanishing) perturbations are
proved not to destroy the ensured stability property for the nominal system [9]. Still, the properties
established in [7,8] are not enough for most real-world applications and we need to go further by ensuring
stronger stability properties (input-to-state stability and Lp stability) for general nonlinear systems and cost
functions. This will allow quantifying the closed-loop system robustness, thereby clarifying the link between
the choice of the cost function and the system robustness properties and will open the door to the study of
interconnected systems.
References
[1] M. Ha, D. Wang, D. Liu (2021). Generalized value iteration for discounted optimal control with stability
analysis. Systems & Control Letters, 147, 104847.
[2] A. Heydari (2017). Stability analysis of optimal adaptive control under value iteration using a stabilizing
initial policy. IEEE Transactions on Neural Networks and Learning Systems, 29(9), 4522-4527.
[3] M. Granzotto, R. Postoyan, L. Buşoniu, D. Neić, J. Daafouz, J. (2020). Finite-horizon discounted optimal
control: stability and performance. IEEE Transactions on Automatic Control, 66(2), 550-565.
[4] A. Al-Tamimi, F.L. Lewis, M. Abu-Khalaf (2008). Discrete-time nonlinear HJB solution using approximate
dynamic programming: Convergence proof. IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics), 38(4), 943-949.
[5] Y. Jiang, Z.-P. Jiang (2017). Robust Adaptive Dynamic Programming. John Wiley & Sons.
[6] B. Pang, T. Bian, Z.-P. Jiang (2022). Robust policy iteration for continuous-time linear quadratic regulation.
IEEE Trans. on Automatic Control, 67(1), 504-511.
[7] M. Granzotto, O.L. De Silva, R. Postoyan, D. Nesic, Z.-P. Jiang (2022). Policy iteration: for want of recursive
feasibility, all is not lost. arXiv preprint arXiv:2210.14459.
[8] M. Granzotto, R. Postoyan, L. Buşoniu, D. Neić, J. Daafouz. (2020). Finite-horizon discounted optimal
control: stability and performance. IEEE Transactions on Automatic Control, 66(2), 550-565.
[9] C.M. Kellett, A.R. Teel (2004). Smooth Lyapunov functions and robustness of stability for difference
inclusions. Systems & Control Letters, 52(5), 395-405.
[10] S. Zoboli, V. Andrieu, D. Astolfi, G. Casadei, J.S. Dibangoye, M. Nadri. (2021, December). Reinforcement
learning policies with local LQR guarantees for nonlinear discrete-time systems. In 2021 60th IEEE Conference
on Decision and Control (CDC) (pp. 2258-2263).
[11] S. Zoboli, D. Astolfi, V. Andrieu, V. (2023). Total stability of equilibria motivates integral action in discrete-
time nonlinear systems. Automatica, 155, 111154.
Keywords:
Control engineering, dynamic programming, Lyapunov stability, robustness, dissipativity, input-to-st
Conditions:
Duration: 3 years
Location: CRAN, ENSEM - Vandoeuvre-lès-Nancy, France
Location: CRAN, ENSEM - Vandoeuvre-lès-Nancy, France
Department(s):
| Control Identification Diagnosis |
Funds:
ANR OLYMPIA
Publications: