RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion

Supplementary video for “RL Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion“ by Dongho Kang, Jin Cheng, Miguel Zamora, Fatemeh Zargarbashi, and Stelian Coros. Abstract: This letter presents a versatile control method for dynamic and robust legged locomotion that integrates model-based optimal control with reinforcement learning (RL). Our approach involves training an RL policy to imitate reference motions generated on-demand through solving a finite-horizon optimal control problem. This integration enables the policy to leverage human expertise in generating motions to imitate while also allowing it to generalize to more complex scenarios that require a more complex dynamics model. Our method successfully learns control policies capable of generating diverse quadrupedal gait patterns and maintaining stability against unexpected external perturbations in both simulation and hardware experiments. Furthermore, we demonstrate the adaptability of our method to more complex locomotion tasks on uneven terrain without the need for excessive reward shaping or hyperparameter tuning. Acknowledgement: We express our gratitude to Zijun Hui for his assistance with the robot experiments. This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme. Computational Robotics Lab: Dongho Kang - - - Stelian Coros -