Align-RUDDER: reward redistribution from few demonstrations
The first write-up of Align-RUDDER: two modifications to RUDDER that let profile models built from as few as two demonstrations carry the reward redistribution.
Reinforcement Learning algorithms require a large number of samples to solve complex tasks with sparse and delayed rewards. Complex tasks can often be hierarchically decomposed into sub-tasks. A step in the Q-function can be associated with solving a sub-task, where the expectation of the return increases. RUDDER has been introduced to identify these steps and then redistribute reward to them, thus immediately giving reward if sub-tasks are solved. Since the problem of delayed rewards is mitigated, learning is considerably sped up. However, for complex tasks, current exploration strategies as deployed in RUDDER struggle with discovering episodes with high rewards. Therefore, we assume that episodes with high rewards are given as demonstrations and do not have to be discovered by exploration. Typically the number of demonstrations is small and RUDDER’s LSTM model as a deep learning method does not learn well. Hence, we introduce Align-RUDDER, which is RUDDER with two major modifications. First, Align-RUDDER assumes that episodes with high rewards are given as demonstrations, replacing RUDDER’s safe exploration and lessons replay buffer. Second, we replace RUDDER’s LSTM model by a profile model that is obtained from multiple sequence alignment of demonstrations. Profile models can be constructed from as few as two demonstrations as known from bioinformatics. Align-RUDDER inherits the concept of reward redistribution, which considerably reduces the delay of rewards, thus speeding up learning. Align-RUDDER outperforms competitors on complex artificial tasks with delayed reward and few demonstrations. On the MineCraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently. Code is published on GitHub
The paper is on arXiv.
Read next
- Align-RUDDER: learning from few demonstrations by reward redistribution Replacing RUDDER's LSTM with a profile model from multiple sequence alignment, so reward redistribution works from a handful of demonstrations.
- Recap of the 2nd International RL Bootcamp A look back at the RL Bootcamp 2025 in Salzburg: highlights, speakers, and the recordings, slides and community links the organizers published.
- InfODist: online distillation with informative rewards Why curriculum-learning agents generalise badly to the next task, and why the culprit is the states they explore rather than the non-stationarity they endure.