Efeito do ruído de Prova no Controle Ótimo LQR via Q-Learning baseado em filtragem adaptativa

Abstract

Q-learning is a reinforcement learning (RL) method, model-free, that is used to solve the optimal control problem based on learning the action value function (or function Q). The usual way to learn the action value function is to solve a Bellman equation. In this thesis, to solve the Bellman equation in the LQR optimal control problem, an adaptive filtering algorithm based on the normalized least-mean-square (NLMS) algorithm is used instead of the recursive least-squares (RLS). A general requirement for achieving convergence in adaptive filtering algorithms is the excitation persistence condition. The persistence of excitation is a condition imposed so that the matrix formed by the regressor vectors has all columns linearly independent. In the context of optimal control via Q-learning, persistence of excitation is obtained by adding a probe noise to the control action. The probe noise affects real system states and may affect the performance of the adaptive filter in solving the Bellman equation. In this work, a study is carried out on the effect of probe noise based on the covariance matrices of the states and control inputs of the system, where a closed formula and convergence properties of such matrices are obtained. Furthermore, it is verified through numerical experiments that the NLMS algorithm presents superior performance when compared to the RLS algorithm, in cases where the probe noise has small variance. The use of the NLMS algorithm in our approach has two advantages: first, the NLMS algorithm presents lower computational complexity when compared to the RLS algorithm; the second, to obtain the persistence of the excitation condition, one can use probe noises with low variance, which is desirable in real-world applications.