Reinforcement Learning

What a participant can do afterwards.

• Formulate a problem as an MDP with explicit observations, states, actions, transitions, rewards, termination and truncation.
• Explain returns, value functions, action values, Bellman equations, policies, advantages and the exploration-exploitation trade-off.
• Implement value iteration, SARSA and Q-learning and compare on-policy with off-policy behaviour.
• Derive and implement a minimal REINFORCE agent and explain variance reduction, actor-critic methods and generalized advantage estimation.
• Train and evaluate PPO and a continuous-control agent using maintained libraries.
• Design reward functions, identify reward hacking and run reward or hyperparameter ablations across multiple seeds.
• Explain the relationship between RL, feedback control, LQR/MPC, offline RL, safe RL and modern LLM post-training.
• Design a verifiable reward function and interpret a small RLVR training loop without a large distributed GPU job.

Realistic depth within three days.

The difference between these three columns is the difference between a course description and a course. Everything in the third column is real content — it is simply not something anyone builds from scratch in 21 hours.

MDPs, Bellman reasoning and tabular control

Establish the mathematical objects and implement core temporal-difference learning.

1. 09:00

   Orientation and problem framing

   Sequential decisions, agent-environment interaction, trajectories and examples from robotics, scheduling and resource allocation.

2. 09:30

   Markov decision processes

   States, observations, actions, transition dynamics, rewards, policies, the Markov property, partial observability, termination and truncation.

3. 10:45

   Values and Bellman equations

   Discounted return, V and Q functions, Bellman expectation and optimality equations, policy evaluation, value and policy iteration.

4. 13:15

   Control-theory bridge

   Dynamical systems, feedback, open- and closed-loop control, stability, PID, LQR and MPC as complementary approaches.

5. 14:15

   Monte Carlo and TD learning

   Bootstrapping, bias and variance, TD error, eligibility intuition and update targets.

6. 15:15

   SARSA and Q-learning

   On-policy versus off-policy control, exploration policies and why learned behaviour differs in risky environments.

7. 16:00

   Lab 1 — tabular RL Lab

   Implement value iteration, SARSA and Q-learning in a small Gymnasium environment and compare policies and learning curves.

Day output Working tabular implementations and a short analysis of convergence, exploration and on-policy/off-policy behaviour.

Policy gradients, PPO and continuous control

Move from value tables to neural policies and practical training pipelines.

1. 09:00

   Recap and diagnostics

   Review Bellman and TD concepts and inspect common learning-curve failure patterns.

2. 09:30

   Function approximation and DQN

   Neural value functions, replay buffers, target networks, instability and the role of DQN as a bridge to deep RL.

3. 10:45

   Policy gradients and REINFORCE Lab

   Log-derivative trick, stochastic policies, return-weighted updates, baselines and a minimal implementation.

4. 13:15

   Actor-critic and advantages

   Critics, temporal-difference advantages, generalized advantage estimation, entropy and the bias-variance trade-off.

5. 14:15

   Proximal Policy Optimization

   Clipped objectives, rollout batches, epochs, value loss, entropy bonus, diagnostics and implementation details.

6. 15:15

   Reward and experiment design

   Sparse and dense rewards, potential-based shaping, reward hacking, normalization, seeds, wrappers and confidence intervals.

7. 16:00

   Lab 2 — PPO and continuous control Lab

   Train PPO on a discrete task and PPO or SAC on Pendulum; compare stability, sample use, returns and policy entropy.

Day output A minimal REINFORCE implementation, a framework PPO/SAC run and a reproducible experiment configuration.

Evaluation, safe RL and modern LLM post-training

Evaluate policies rigorously and connect classical RL to current verifiable-reward training.

1. 09:00

   RL evaluation

   Training versus evaluation episodes, multi-seed confidence, generalization, robustness, checkpoint selection and reproducibility.

2. 09:45

   Continuous-control choices

   PPO versus SAC and TD3, deterministic and stochastic policies, action bounds and sample efficiency.

3. 10:45

   Advanced RL landscape

   Offline, imitation, model-based and multi-agent RL: objectives, data assumptions and when each becomes relevant.

4. 11:30

   Constrained and safe RL

   Constraints, shields, risk-sensitive objectives, unsafe exploration and reward audits.

5. 13:15

   RL for language models

   SFT, preference data, reward models, PPO, DPO context, GRPO-style group baselines and agentic rollouts.

6. 14:30

   RLVR fundamentals

   Verifiable rewards, deterministic checkers, sparse signals, verifier errors, reward hacking and suitable domains.

7. 15:15

   Lab 3 — verifier practicum Lab

   Implement arithmetic or code reward functions, score grouped rollouts and inspect normalized advantages and training diagnostics.

8. 16:15

   Capstone completion Capstone

   Finish environment, baseline, trained policy, reward ablation and multi-seed evaluation.

9. 17:00

   Demonstrations and review Capstone

   Present policies, evidence, failure modes and decisions about the next algorithm or experiment.

Day output A completed RL capstone plus a small RLVR verifier and rollout-analysis notebook.

Four deliverables, each one the input to the next.

01

Tabular control Value iteration, SARSA, Q-learning, exploration comparison and policy visualization.

02

Deep policy learning Minimal REINFORCE plus PPO and continuous-control runs with reproducible configurations.

03

RLVR verifier Custom verifiable reward functions, grouped rollout scoring and advantage diagnostics.

04

Capstone experiment Custom or modified environment, baseline, trained policy, ablation and multi-seed evaluation.

Capstone specification

• Explicit observation, action, transition, reward, termination and truncation definitions.
• A non-learning baseline such as random, heuristic or simple feedback control.
• At least one trained RL policy using an algorithm appropriate to the action space.
• One reward, exploration or hyperparameter ablation and an explanation of the expected mechanism.
• Evaluation across at least three seeds with mean, spread and failure cases.
• A reward-hacking and safety analysis, including what the policy could exploit.
• Saved configuration, learning curves, final policy and a short experiment report.

The tools, and why each one is in the room.

Environment API

Gymnasium Standard single-agent environment interface and small reference tasks.

Foundations

NumPy + PyTorch Used for tabular methods and the minimal REINFORCE implementation.

Reliable baselines

Stable-Baselines3 Maintained PPO, DQN, SAC and TD3 implementations for practical training.

Research components

TorchRL PyTorch-native collectors, buffers, objectives and modular research building blocks.

Distributed / multi-agent

Ray RLlib Discussed for scaling, multi-agent systems and offline data workflows.

LLM post-training

Hugging Face TRL Accessible custom reward functions and GRPO/PPO-style trainers for workshop-level examples.

Scale-out RLVR

verl or OpenRLHF Surveyed for distributed, production-oriented LLM reinforcement learning.

Discussed, not taught hands-on

• CleanRL is useful for readable single-file reference implementations, but the workshop already includes from-scratch tabular and REINFORCE code.
• PettingZoo is introduced when the participant group specifically needs multi-agent environments.
• MuJoCo tasks are optional: installation and compute variability can consume too much workshop time, and Pendulum is a reliable continuous-control baseline.