Agentic AI for Scientific Discovery — the RL Bootcamp 2026 slides

The 3rd International Reinforcement Learning Bootcamp runs from 16 to 18 September 2026 at Paris Lodron University Salzburg, hosted by the SARL group at the IDA Lab. It starts at first principles and works up to current methods over three days of lectures, workshops and coding sessions, and — as in previous years — participation is free of charge. The second edition drew over a hundred participants from twelve countries, and closed by pointing at this one.

I am speaking on the Friday morning. The talk is called Agentic AI for Scientific Discovery, and it asks a narrower question than the title suggests: when implementation stops being the bottleneck, which experiment becomes possible — and how do you decide whether it worked?

The deck is below. It is the real thing, not an export: click through it here, or open it as its own page and present from it.

Open the deck — loads Agentic AI for Scientific Discovery — RL Bootcamp 2026 in an embedded viewer

← → move · F fullscreen · X slide index · N speaker notes · B backup slides

What the talk argues

Three claims, in order.

A model and its harness are one system. An agent is a policy plus the scaffolding that selects its context, mediates its tool calls and decides when to stop. Change the scaffolding and you change the measured capability, which is why a benchmark number without its harness is not a model result. The first third of the deck is spent on the parts of that scaffolding that turn a long-running agent into usable training evidence: durable state across context windows, recoverable task state, evaluation given its own job, and credit assignment over tool-use trajectories.

Reinforcement learning starts where the target token stops existing. Pre-training and supervised fine-tuning both have a target token — the next one in the corpus, or the one a demonstrator wrote — so the objective is a cross-entropy against it. A preference stage fits a scalar from comparisons; a reasoning stage computes one with a program. In both cases the update observes only a score over the complete response, so per-token credit has to be estimated, and every named objective in the literature is a choice about that estimate.

The acronym zoo is three knobs, not a dozen algorithms. The estimator has not changed much since PPO. What later methods change is the baseline (a learned value network, a leave-one-out mean, a group mean), the ratio (token-level, sequence-level, truncated) or the infrastructure (the mismatch between the engine that generated a token and the engine that scores it). Outcome-GRPO uses same-prompt group rewards to standardize advantages without a learned critic, while retaining a clipped policy surrogate. Reference KL controls drift in both RLHF/PPO and GRPO: the deck shows it inside PPO’s shaped rewards and explicitly in GRPO’s objective, not as a replacement for the critic’s variance reduction. The full original GRPO algorithm sits beside that comparison, followed by what a group baseline costs — the completion barrier it imposes, and what an asynchronous single-rollout scheme trades away to remove it.

There is a fourth thread running underneath, which is really why I was asked to give this talk: what counts as evidence. A reward that checks a final answer accepts a response whose derivation is nonsense, so one slide shows exactly that — a correct answer with an invalid certificate, accepted by answer-only scoring and rejected once the certificate is checked.

Two slides from my exploratory manuscript, What Must Experience Models Preserve?, make that distinction concrete. Interactive mathematical tiles expose the assumptions behind learner-relative guarantees. Measured learning curves show both improvement and failure; a looping CartPole replay illustrates two saved policies, explicitly separated from the manuscript’s evaluations and larger pilot. These slides sit immediately before the research-claim-to-experiment walkthrough.

How to read the deck

Forty-four timed slides make a 44-minute, 30-second route, including two one-minute discussion pauses. The scientific-discovery timeline follows the Navier–Stokes example: hover or focus a milestone for a preview, then click or press Enter to enlarge its source figure and scope. The five mathematical tiles on slide 37 use the same interaction. Dentate’s workbench introduction now precedes recurrent depth and parameter sharing. Behind the main route sit 101 untimed backup slides: the derivations for each named objective, the exact local implementation choices, the evaluation protocol, and the original teaching material from earlier editions, kept with corrected citations rather than rewritten.

• ← → or Space walk the whole deck, backups included.
• X opens a searchable index — type a method name rather than remembering a slide number.
• B jumps to the backups and back to where you were.
• N shows the speaker notes, each with its own clock, so the deck also reads as a script.
• F is fullscreen, and P prints the complete deck.

Every slide that carries a number carries its source in the footer, and the distinctions the field usually blurs are kept explicit on the slide: author-reported demonstrations are labelled as such, my own unpublished experiments are labelled as mine, and forecasts are labelled as conditional. A slide that says “this is not a controlled result” is doing more work than one that quietly implies it is.

The runnable half

The talk ends on something attendees can run rather than admire. Dentate is the small training module the last third uses as its worked example: a recurrent model with about a million parameters in its shared core (7.3M in total), supervised fitting on a verifiable task, a support gate, then GRPO initialised from that exact checkpoint, then evaluation on a frozen task set.

pip install "dentate[demo]"
dentate demo init
dentate demo run --out my-experiment
dentate serve   # http://127.0.0.1:8793

It runs offline on a CPU after demo init, and there is a Colab notebook for anyone who would rather not install anything. The point of the exercise is not the score — a bounded starter run will often fail its support gate, and that is a result, not a bug. The point is that the whole loop, including the gate that refuses to proceed, is inspectable on a laptop.

Where the recordings land

The organisers publish on YouTube as Reinforcement Learning — that is where to look for this year’s sessions once they are cut. My talk from the 2025 edition, Reinforcement Learning — Where We Are and What’s Next, is already up.

If you are anywhere near Salzburg this week: three days, free, and the fastest route I know from “I have read about policy gradients” to “I have trained an agent and know why the number moved”. Registration and the full programme are on the event page.

Deck last rebuilt 18 September 2026. Figures reproduced from published papers keep their original attribution and licence; the two external demonstration clips are short excerpts of author-reported material, credited on the slide. Slide numbers in the speaker notes refer to this build.