Reactive exploration to cope with non-stationarity in lifelong reinforcement learning
Tracking continual domain shifts as they happen: why policy-gradient methods adapt faster than Q-learning when the environment keeps moving.
Abstract
In lifelong learning, an agent learns throughout its entire life without resets, in a constantly changing environment, as we humans do. Consequently, lifelong learning comes with a plethora of research problems such as continual domain shifts, which result in non-stationary rewards and environment dynamics. These non-stationarities are difficult to detect and cope with due to their continuous nature. Therefore, exploration strategies and learning methods are required that are capable of tracking the steady domain shifts, and adapting to them. We propose Reactive Exploration to track and react to continual domain shifts in lifelong reinforcement learning, and to update the policy correspondingly. To this end, we conduct experiments in order to investigate different exploration strategies. We empirically show that representatives of the policy-gradient family are better suited for lifelong learning, as they adapt more quickly to distribution shifts than Q-learning. Thereby, policy-gradient methods profit the most from Reactive Exploration and show good results in lifelong learning with continual domain shifts. Our code is available at: github.com/ml-jku/reactive-exploration.
The paper is on arXiv.
Read next
- Recap of the 2nd International RL Bootcamp A look back at the RL Bootcamp 2025 in Salzburg: highlights, speakers, and the recordings, slides and community links the organizers published.
- InfODist: online distillation with informative rewards Why curriculum-learning agents generalise badly to the next task, and why the culprit is the states they explore rather than the non-stationarity they endure.
- XAI and strategy extraction via reward redistribution Using Align-RUDDER as an interpretability method: multiple sequence alignment surfaces the key events an agent relies on, and those events are usually human-readable sub-tasks.