Modelagem não linear e controlador ator-crítico para manobrabilidade de um USV baseado em DLQT-I e programação dinâmica heurística dependente de ação

Abstract

Oil spills cause environmental degradation, socio-economic problems, and severe harm to human health. Due to the challenges of monitoring such events over large areas, the use of Unmanned Surface Vehicles (USVs) has become essential for this task. The application of these devices requires equipping them with sensors for water quality analysis, as well as a Guidance, Navigation, and Control (GNC) system. Specifically, to enable maneuverability, this work presents the development of an online control system proposal based on Adaptive Dynamic Programming (ADP) and Reinforcement Learning (RL). The development of a control system for USV maneuverability presents implementation challenges due to the difficulties in system identification for mathematical modeling, as it involves a complex, nonlinear, coupled model with uncertainties. External disturbances and possible parametric variations further increase the obstacles to its design. In this context, this dissertation presents the design of an adaptive optimal controller based on data for controlling the speeds of a fully actuated USV, ensuring maneuverability. The methodology developed for designing the online optimal controller integrates ADP and RL within the Action Dependent Heuristic Dynamic Programming (ADHDP) technique, aiming to compute the implicit online solution of the Hamilton-Jacobi-Bellman (HJB) equation in the form of the Discrete Algebraic Riccati Equation (DARE). The system employs Reinforcement Learning using an actor-critic structure with an optimal control approach to compute the optimal control law based on control action data and process states. The developed controller is a Discrete Linear Quadratic Tracker with integral action (DLQT-I), referred to as ADHDP-DLQT-I, and its results are evaluated using a nonlinear model simulator solved via the Runge-Kutta method. The performance of the ADHDP-DLQT-I system is compared with an adaptive PID controller tuned using the Least Mean Square (LMS) method.