5 useful things you'll learn in my new post-training textbook (shipping now!)

What it claims

The book (Manning, Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs) is pitched as a foundations-and-intuitions text, not a code cookbook. The author says it is not a beginner book: it assumes roughly a completed CS bachelor's degree. Its distinctive value is coverage of post-training methods the author found had little or no other online explanation, naming rejection sampling, outcome reward models and character training. The title is described as "a little outdated," but the content is current. On-policy distillation was added at the last minute.

How it works

Structure and emphasis, as the page gives them:

• About 25% of the book is RL by page count. It runs from the policy-gradient theorem through PPO to GSPO and CISPO. The stated goal is to let readers judge whether a newly announced algorithm is "fake or has potential."
• PPO clipping is reduced to six regions of the surrogate objective. There are two gradient cases (positive or negative token advantage), each depending on the policy ratio. On the first gradient step of a batch the ratio is 1 and gradient always flows. After the policy updates, the gradient either stays the same or becomes 0, and that cutoff is what the clipping arguments control.
• Systems framing. Modern RL is treated as balancing three things: how off-policy the data is, training-inference mismatch, and throughput. The baseline design is asynchronous, with separate learner and actor GPUs. Agentic tasks are presented as extra infrastructure on top of this design, not a different paradigm. Implementation starts from the general form pg_loss = -advantages * ratio.
• Chapter 2 covers history in three eras:
• RL on preferences generally, until about 2018
• applying it to language models, 2019–2022
• post-ChatGPT practice, 2023 onward
• Chapters 10–12 cover opaque data-industry practices.
• Chapter 12 is on distillation. It traces the path from 2015-era knowledge distillation to multi-teacher on-policy distillation (MOPD) via "2-3 key insights."
• The second half covers over-optimization, regularization, evaluation and character training. This includes a math-level account of why RL generalizes while SFT forgets, and how frontier labs shape model personality and why that often goes too far.

How to use it

• The full text is free online. It comes with a 12-hour course (slides plus YouTube video), a simple codebase with suggested exercises, and model completion comparisons.
• The print edition ships from Manning and Amazon US now, and from Amazon UK in October.
• The post offered 50% off at Manning with code PBLambert, valid only until August 19.

Limits

• This is a launch post, so it shows no exercise content, code or chapter text beyond one formula and a description of the PPO figure.
• The author describes the explanatory text as "higher voice than your average textbook," because it is partly reworked from Interconnects blog posts.
• Readers without LLM/RL background should expect a steep start. Readers wanting a list of code recipes should note the author positions the book against that format.