Day 4: Policy Gradients and RL in the Real World

Learning the policy directly, from REINFORCE to PPO and GRPO, then where RL runs in the real world: RLHF for language models, multi-agent RL and robotics.

Topic Description & Objectives Resource
Policy Gradients and RL in the Real World REINFORCE, PPO and GRPO (Lab 4), then RLHF (Lab 5), multi-agent RL and robotics Open
Lab 4: GRPO on Flappy Bird Exercise: implement Group Relative Policy Optimization in PyTorch and train it to play Flappy Bird, then put the critic back to get PPO and race the two on the same budget Notebook Colab
Lab 5: RLHF from Scratch Exercise: fit a reward model to pairwise preferences, optimise it with GRPO under a KL leash, and watch the policy game the reward when the leash comes off Notebook Colab
Optional: GRPO on Flappy Bird (Solution) Worked solution to the GRPO and PPO exercise Notebook Colab
Optional: RLHF from Scratch (Solution) Worked solution to Lab 5 Notebook Colab
Optional: REINFORCE on CartPole Exercise: policy gradients from scratch, then the causality trick and a baseline, and how each one cuts the variance of the gradient Notebook Colab
Optional: REINFORCE on CartPole (Solution) Worked solution to the REINFORCE exercise Notebook Colab
Optional: DPO on a Tiny Language Model Exercise: preference-tune a tiny GPT with DPO, with no reward model and no RL loop, then read the implicit reward off the policy Notebook Colab
Optional: DPO on a Tiny Language Model (Solution) Worked solution to the DPO exercise Notebook Colab
Optional: Continuous Control with PPO Worked lab: PPO with a Gaussian policy on MountainCarContinuous. Plain PPO learns to do nothing; energy-based reward shaping that keeps the best policy gets it to about 94, above the solved bar of 90 Notebook Colab