Each feed is checked for new posts. Every post is read, and those that belong here join the catalog. What Trove read and did not keep stays on file, not shown.
Building Reasoning Models from the Ground Up is a series of eight coding videos (about 11 hours) by Sebastian Raschka, accompanying his book Build a Reasoning Model (From Scratch). Starting from a pre-trained Qwen3 0.6B model, it implements KV-cached generation, a MATH-500 verifier, self-consistency and self-refinement, simplified GRPO for RLVR, and distillation from DeepSeek R1 answers in Python and PyTorch.
Sebastian Raschka surveys the history of text classification — bag-of-words with naive Bayes/logistic regression, RNNs and CNNs with word embeddings, then transformer approaches (BERT/ModernBERT, GPT-style decoder fine-tuning, T5 encoder-decoder) — before analyzing TypeSafe AI's new Jev model, a fast/cheap general-purpose classifier accessed via Choice/Noul/Score APIs. He benchmarks Jev against a fine-tuned ModernBERT on the IMDb sentiment dataset (Jev ~96.2-96.5% vs ModernBERT ~95%), explains how to retrofit a Jev-like classification head onto BERT/GPT models, and discusses probability calibration, including the RLCR (Reinforcement Learning with Calibration Rewards) technique from a 2025 paper that likely inspired Jev's proprietary "RLCD" training method.
Sebastian Raschka's deep-dive on OpenAI's GPT-6 Astra release, explaining the "looped transformer" / recurrent-depth architecture concept rumored to underlie it, tracing it from 2018's Universal Transformer through Nanbeige4.2-3B, ByteDance's Ouro, and Mixture-of-Recursions, and evaluating claims that looping "hides" chain-of-thought reasoning. It also covers Astra's computer-use training via macOS environments on Mac hardware and reviews recent papers (Latent Reasoning recurrent-depth, Beyond Parameters, SMELT, Full-bandwidth transformer) on compute-matched looped-transformer scaling.
Sebastian Raschka (author of "Build a Large Language Model From Scratch") breaks down, in a 48-minute lecture with transcript, exactly how Anthropic's newly announced Claude text watermarking works: it is applied not inside the model but at the sampling stage, using a secret key plus the last few tokens to derive a deterministic random seed. He explains the underlying "tournament sampling" technique (derived from Google's SynthID-Text, published in Nature) where multiple random watermarking functions (G1...Gn) generate bit signatures for candidate tokens that compete pairwise, letting Anthropic later score arbitrary text for the watermark without rerunning the LLM. The piece also covers why the watermark is theoretically removable by editing enough token positions, since only some positions have near-equally-likely alternative tokens.
Sebastian Raschka walks through building an AI-text detector from scratch, similar to Pangram-style models behind Substack's new AI-detection UI feature, by fine-tuning a DistilBERT classifier that outputs a 0-100 AI-generated probability score. He then uses that classifier as a verifier/reward signal to train a small language model (SLM) to produce text that evades the detector, framing the whole project as a case study in building scorers/verifiers for LLM applications beyond math/code reasoning tasks. The piece covers detector approaches broadly (supervised classifiers, perturbation-based tests, perplexity, watermarking) before diving into the DistilBERT fine-tuning implementation.
Sebastian Raschka explains how modern reasoning LLMs implement adjustable "reasoning effort" or thinking-budget controls, tracing the mechanics from DeepSeek-R1's original RLVR recipe through hybrid think/no-think switches (Qwen3) to full effort-level systems in GPT-5.6 and gpt-oss. He then does a comparative deep dive into six flagship open-weight models' documented recipes — DeepSeek V4 (separate effort specialists distilled together), Nemotron 3 Ultra (SFT + truncated-trace training + hard budgets), Kimi K2.5/K3 (Toggle alternating budgeted/unconstrained RL, multi-teacher distillation), GLM-5 (turn-level/interleaved thinking via SFT), Qwen3 (Thinking Mode Fusion), and Inkling (continuous 0-1 effort conditioning during RL) — showing the shared techniques (effort-labeled SFT data, mode-conditioned RL with variable length penalties/context windows, and robustness training for hard token cutoffs) behind these UI toggles.
magazine.sebastianraschka.comcourse
