XAI and strategy extraction via reward redistribution
Using Align-RUDDER as an interpretability method: multiple sequence alignment surfaces the key events an agent relies on, and those events are usually human-readable sub-tasks.
Abstract
In reinforcement learning, an agent interacts with an environment from which it receives rewards, that are then used to learn a task. However, it is often unclear what strategies or concepts the agent has learned to solve the task. Thus, interpretability of the agent’s behavior is an important aspect in practical applications, next to the agent’s performance at the task itself. However, with the increasing complexity of both tasks and agents, interpreting the agent’s behavior becomes much more difficult. Therefore, developing new interpretable RL agents is of high importance. To this end, we propose to use Align-RUDDER as an interpretability method for reinforcement learning. Align-RUDDER is a method based on the recently introduced RUDDER framework, which relies on contribution analysis of an LSTM model, to redistribute rewards to key events. From these key events a strategy can be derived, guiding the agent’s decisions in order to solve a certain task. More importantly, the key events are in general interpretable by humans, and are often sub-tasks; where solving these sub-tasks is crucial for solving the main task. Align-RUDDER enhances the RUDDER framework with methods from multiple sequence alignment (MSA) to identify key events from demonstration trajectories. MSA needs only a few trajectories in order to perform well, and is much better understood than deep learning models such as LSTMs. Consequently, strategies and concepts can be learned from a few expert demonstrations, where the expert can be a human or an agent trained by reinforcement learning. By substituting RUDDER’s LSTM with a profile model that is obtained from MSA of demonstration trajectories, we are able to interpret an agent at three stages: First, by extracting common strategies from demonstration trajectories with MSA. Second, by encoding the most prevalent strategy via the MSA profile model and therefore explaining the expert’s behavior. And third, by allowing the interpretation of an arbitrary agent’s behavior based on its demonstration trajectories.
The chapter is published in xxAI - Beyond Explainable AI.
Read next
- Align-RUDDER: learning from few demonstrations by reward redistribution Replacing RUDDER's LSTM with a profile model from multiple sequence alignment, so reward redistribution works from a handful of demonstrations.
- Agentic AI for Scientific Discovery — the RL Bootcamp 2026 slides The 3rd International RL Bootcamp runs 16–18 September 2026 in Salzburg, free to attend. Here is the deck for my Friday talk, embedded to click through in full, plus what it argues and where the recordings land.
- Recap of the 2nd International RL Bootcamp A look back at the RL Bootcamp 2025 in Salzburg: highlights, speakers, and the recordings, slides and community links the organizers published.