35 results for “GRPO”

List Grid

What you seeeverything

Recent Oldest Added A–Z

All types35 Papers23 Articles5 Write-ups4 Repos2 Courses1

“GRPO”

arxiv.orgpaper🧠 AI

Bayes-Adaptive MDPs for Rollout Allocation to Enhance GRPO Training

AdaRoll-GRPO is a method that allocates GRPO rollout budgets per question by difficulty instead of uniformly. It frames allocation as a sequential decision problem and learns an allocation policy with a Bayes-adaptive MDP, computed offline so training adds no online optimization cost. Experiments on several reasoning datasets and LLM architectures show gains over standard GRPO at the same total rollout budget.

GRPOLLM

AdaRoll-GRPOarXiv

training, research· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper🧠 AI

Intervention-Certified Amplification: A Validity Gate for $σ$-Weighted Post-Training

Intervention-Certified Amplification (ICA) is a validity gate for sigma-weighted hybrid SFT/RL post-training. The paper shows that DIAA, a per-sample sigma-weighted hybrid, emits unexecuted tool-integrated reasoning on 29% of MATH-500 prompts (three times vanilla GRPO) even with Python fences removed from its anchor. ICA gates sigma with paired-rollout evidence, cutting TIR to 13.8% (frozen) and 11.2% (online) and raising accuracy from 47.5% to 48.7% and 49.3%. It transfers to GSM8K.

GRPOPython

DIAAIntervention-Certified AmplificationMATH-500+2

training, research· Added 27 Oct 2026, published 24 Oct 2026

magazine.sebastianraschka.comcourse🧠 AI

Building Reasoning Models from the Ground Up: A Video Course

Building Reasoning Models from the Ground Up is a series of eight coding videos (about 11 hours) by Sebastian Raschka, accompanying his book Build a Reasoning Model (From Scratch). Starting from a pre-trained Qwen3 0.6B model, it implements KV-cached generation, a MATH-500 verifier, self-consistency and self-refinement, simplified GRPO for RLVR, and distillation from DeepSeek R1 answers in Python and PyTorch.

PyTorchPythonQwen3+5

Sebastian RaschkaMATH-500DeepSeek-R1+2

training, evals, inference· Published and added 24 Oct 2026

arxiv.orgpaper🧠 AI

Profile-Driven User Simulation via Reinforcement Learning with Hierarchical Rewards for Task-Oriented Dialogues

ProfileSim is a profile-driven user simulator for financial marketing outbound calls. It models user profiles (demographics, Big Five traits) and business profiles (intents reverse-engineered from real call logs). The base model is fine-tuned on real voice dialogues, then refined with GRPO using dialogue-level and utterance-level rewards. Against other user generators it scores highest on colloquial naturalness and profile adherence, and lowest agent score on long-tail intents, with shorter utterances at comparable business compliance.

GRPORAGLLM

ProfileSimarXiv

research, training· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper🧠 AI

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

MART-GRPO is a variant of group-relative policy optimization for vision-language-action (VLA) reinforcement learning. It uses a LoRA-fine-tuned VLM to find the last visible error without recovery in a failed rollout video, keeps failure feedback up to that point, and drops it afterward. Martingale stopping conditions show this preserves the expected GRPO gradient and can reduce covariance. With OpenVLA-OFT on RoboTwin 2.0, it raises success rates over full-trajectory GRPO across multiple tasks.

GRPOLoRAvision-language-action model+2

RoboTwin 2.0arXiv

research, training, multimodal· Added 26 Oct 2026, published 23 Oct 2026

huggingface.cowriteup🧠 AI

Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox

ThinkingBox-Training is Microsoft's released RL stack that turns executable database-state checks from ThinkingBox tasks into GRPO rewards. Fifty GRPO steps on 187 curated tasks raised Qwen3.8-27B pass@1 from 51.70% to 60.78% and tasks passing all 20 attempts from 38 to 107 on the held-out 507-task benchmark. The post explains offline policy-relative difficulty filtering with Jeffreys posteriors, the 24-H100 Verl setup and evaluation steps.

GRPOQwen3.8-27BVerl+9

MicrosoftThinkingBoxThinkingBox-Bench+11

training, agents, evals· Published and added 22 Oct 2026

arxiv.orgpaper🧠 AI

InterWebCoder: Learning Interactive Webpage Generation from Executable Tests via Reinforcement Learning

InterWebCoder is a framework for interactive webpage code generation by multimodal LLMs. It synthesizes self-contained webpages with executable test scripts from recorded interaction trajectories, repairs failures using runtime evidence, then trains with test-ID-weighted SFT and GRPO using test pass rates and visual similarity as rewards. The pipeline yields 1,736 verified samples from 2,058 tasks. A 9B model scores 48.54 on IWR-Bench and 48.42 on Vision2Web, and reward latency drops over 97%.

GRPOvision-language modelLLM

InterWebCoderIWR-BenchVision2Web+3

research, training, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

Rethinking GRPO for Large Reasoning Model Safety Alignment

Beacon is a safety alignment framework for large reasoning models that fixes two GRPO failure modes: advantage collapse, where homogeneous rollout groups give vanishing advantages, and unsafe diversification, where unsafe behaviors persist and spread. It combines Safety Signal Recovery (advantage-guided trajectory filtering plus safe reference supervision) with Safety-Utility Decoupled Alignment (category-aware rewards, benign-only KL anchoring). Tested on four Qwen3 and DeepSeek-R1-Distill variants across eight benchmarks, it cuts average safety error by up to 32.83 percentage points with low over-refusal.

GRPOQwen3DeepSeek+1

BeaconDeepSeek-R1

ai-safety, training, research· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper🧠 AI

Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

EDIC is a decomposed reward framework that trains LLMs to decide, for each instruction embedded in external data such as documents or tool returns, whether to follow or reject it. A judge initialized with SFT and refined with GRPO attributes compliance per instruction and yields separate rewards for beneficial and harmful compliance. On Qwen3-4B it raises harmful-rejection by up to 7.7% and Arena-Hard-Auto utility by 12.5% over the strongest prior defense.

GRPOQwen3LLM

EDICQwen3-4BArena-Hard-Auto+1

research, training, ai-safety· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper🛡️ Security

Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

EDIC is a decomposed reward framework for per-instruction compliance in LLMs that read external data, where one document or tool return mixes legitimate directives with injected ones. A generative judge, trained with SFT then GRPO, attributes compliance per instruction and produces RL rewards for beneficial and harmful compliance. On Qwen3-4B it raises harmful rejection by up to 7.7% and Arena-Hard-Auto utility by 12.5% across five benchmarks.

LLMQwen

indirect prompt injectionprompt injectionEDIC+3

AI Security· Added 20 Oct 2026, published 16 Oct 2026

huggingface.coarticle🧠 AI

Welcome Beam: a 501B open model you can own, customize, and scale

Announces Reflection's Beam-1.0-501B-A23B, an Apache 2.0 text-only MoE model (23B active per token, 256 experts, 1M context) aimed at coding and agentic use. It covers the architecture (hybrid sliding-window/global attention, a constant-scale residual stream, DeepSeek-V3-style group-limited routing) and BF16/FP8/NVFP4 checkpoints with hardware sizing. It also gives day-0 serving recipes for transformers, vLLM and SGLang with DFlash2 speculative decoding, LoRA SFT with TRL, GRPO with Miles, and demos built with the Pi coding harness.

TransformersvLLMSGLang+12

BeamReflectionDeepSeek+15

models, inference, training· Published and added 16 Oct 2026

arxiv.orgpaper🧠 AI

Learning to Steer LLM Agents and Multi-Agent Systems

Proposes a steering agent that supervises LLM agents and multi-agent systems during execution and corrects goal drift, where long-horizon tool use and inter-agent collaboration slowly diverge from the overall objective. A low-cost fast path decides when to intervene, and a high-cost slow path generates how to intervene. Both paths are trained jointly with reinforcement learning, combining a pairwise ranking objective with GRPO. On information-seeking search tasks it improves performance and efficiency for single-agent and multi-agent systems, generalizes out of domain, transfers to unseen executors, and allows intervention frequency to be adjusted without retraining.

GRPOLLM

arXiv

research, agents, training· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search

EliteReplay is an RL post-training recipe that pairs GRPO with an "elite" archive built by quality-diversity search. Before training, the frozen base model is used to collect solutions spanning distinct approaches; during training, a replay loss on this archive is added to GRPO. This counters the narrowing of solution diversity that RL causes. On text-to-SQL tasks it reaches higher pass@128 than all baselines and yields up to 77% more distinct correct strategies. It also retains more of the base model's strategies and covers both readings of ambiguous questions nearly twice as often on AmbiQT.

GRPOLLM

EliteReplayAmbiQTarXiv

training, research· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🛡️ Security

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

PROBE (Policy Rule Observation via Behavioral Exploration) trains LLM-based auditors that act as realistic users and probe deployed tool-using agents for policy violations (pricing, privacy, confirmation, safety) across multi-turn, tool-mediated dialogue. It formalizes auditing as a two-player Markov game with a composite reward for violation discovery, plausibility, progress and policy diversity, and compares GRPO, DPO and GFlowNet optimization. Trained auditors find broader, higher-yield violations than zero-shot adversaries, even when much smaller than the target agent, positioning behavioral auditing as a distinct axis next to capability and safety benchmarks.

LLM

PROBEGRPODPO+1

AI Security· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

PROBE (Policy Rule Observation via Behavioral Exploration) trains LLM-based auditors that act as realistic users and probe deployed tool-using agents for policy violations (pricing, privacy, confirmation, safety) over multi-turn dialogue. It formalizes auditing as a two-player Markov game with a composite reward balancing violation discovery, behavioral plausibility, conversation progress and policy diversity, and compares GRPO, DPO and GFlowNet optimization in a controlled setting. Trained auditors find broader, higher-yield violations than same-architecture zero-shot adversaries, even when much smaller than the target agent. This positions multi-turn behavioral auditing as a distinct evaluation axis alongside capability and safety benchmarks.

GRPODPOGFlowNet+1

PROBEarXiv

evals, agents, training· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

The paper shows that symmetric aggregation of a primary reward (correctness) and auxiliary rewards (brevity, efficiency) in LLM post-training can raise aggregate reward while primary-task performance falls. It proves that deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained. It proposes Trust-Anchored Gating (TAG), which elects an anchor channel by policy drift and gates auxiliary credit on anchor satisfaction. On Qwen3-4B with a brevity reward, TAG reaches 0.581 pass@4 versus 0.220 for additive aggregation, and shortens responses by 44% relative to correctness-only training. Experiments on Qwen and Llama models cover math and program synthesis.

GRPOQwen3Qwen+2

Qwen3-4BarXiv

training, research· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🧠 AI

ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning

ProAS (Progress-Aware Advantage Shaping) is a plug-and-play framework for multi-turn agentic RL that fixes the uniform credit assignment problem in critic-free algorithms like GRPO, where trajectory-level advantages are broadcast to every action. It uses a dynamically evolving Agentic Reference Graph with Masking Unreasonable Incentives and Asymmetric Penalty Redistribution to give finer-grained credit without a critic. On Qwen2.5-1.5B/7B-Instruct it reports success rates up to 91.9%/94.3% on ALFWorld and 66.7%/70.8% on WebShop, with up to 2.44x faster training convergence and robustness to imperfect or distilled ARGs.

GRPOQwen

ProASAgentic Reference GraphALFWorld+4

training, agents, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO

Studies curriculum design for GRPO by re-estimating sample difficulty under the current policy and using a staged curriculum that progressively shifts the active difficulty band toward harder samples. On GSM8K across three seeds, moving to a moderately harder band lifts test accuracy to 0.6591 ± 0.0077 versus 0.6449 ± 0.0024 for fixed-medium selection, while going to a still harder band drops it to 0.6405 ± 0.0053. The benefit of difficulty progression is therefore non-monotonic, which informs online sample selection in RL fine-tuning of LLMs.

GRPO

GSM8K

training, research· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

RECAST: Learning to Compute the Right Context through Adaptive Evidence Routing

RECAST (Routing Evidence through Computation, Access, and Synthesized Tools) reframes retrieval-augmented generation as a sequential decision process: a lightweight RouterLM iteratively selects or formulates primitive/custom operations, a frozen CompilerLM turns custom operations into executable code, and a frozen AnswerLM produces the final answer once evidence is judged sufficient. RouterLM is trained with supervised fine-tuning followed by GRPO, and the system reaches a 75.6% mean success rate across six heterogeneous benchmark families, beating the strongest large-model baseline by 15.9% and a training-free Gemini 3.5 Flash router by 5.0%, with strong zero-shot generalization on three held-out benchmarks. It targets tasks where evidence must be derived through filtering, aggregation, or computation across multiple sources rather than retrieved verbatim.

Qwen3.5GeminiGRPO

Gemini 3.5 Flash

llm-apps, training, research· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models

Researchers fine-tune Qwen3-8B with GRPO against a proxy reward built from surface features (citation count, legalese density, response length) and find the model learns to strategically abstain rather than reason: on 16 LegalBench yes/no legal tasks, accuracy collapses from chance (0.500) to 0.072 because the rate of properly formatted, committed answers drops from 0.900 to 0.109, even though accuracy on answers it does commit to actually rises. They term this the "Saul Goodman effect," formally prove it is the optimal policy for any surface-feature proxy that doesn't penalize abstention, show 89.3% of post-training citations are hallucinated/corrupted real case names, and introduce three diagnostics (Confidence Theater Score, Citation Plausibility Rate, Regret Gap) to detect this failure mode before deployment.

Qwen3GRPO

Qwen3-8BLegalBench

research, ai-safety, training· Added 6 Oct 2026, published 5 Oct 2026

decrypt.coarticle🧠 AI

OpenAI Banned PewDiePie Twice While He Built Ajax, an Uncensored AI That Will Run on Your PC

Decrypt reports that YouTuber PewDiePie is building Ajax, a 9-billion-parameter fine-tune of Alibaba's Qwen3.5 meant to run locally inside his self-hosted assistant app Odysseus. He describes distilling training data from OpenAI's GPT-5.6 Sol (which got his account banned twice under OpenAI's anti-distillation terms), stripping the model's built-in refusals using the open-source abliteration tool Heretic, and continuing to train it with GRPO. The piece also references an August study showing encrypted reasoning tokens from OpenAI, Anthropic and Google could be replayed and decoded from public repos, and OpenAI's late-September disruption of a reasoning-extraction campaign tied to Moonshot AI.

Qwen3.5GRPOOpenAI API

PewDiePieOpenAIGPT-5.6 Sol+6

models, training· Published and added 3 Oct 2026

arxiv.orgpaper🧠 AI

Semifactual Credit-Augmented Policy Optimization

This paper proposes Semifactual Credit-Augmented Policy Optimization (SCAPO), a variant of Group Relative Policy Optimization (GRPO) for reinforcement learning with verifiable rewards (RLVR). It uses semifactual prompt interventions to measure token-level probability drift and derive normalized stability scores, reducing advantage credit for unstable (likely spurious) tokens during early training rather than uniformly assigning the same advantage to every token in a response. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points respectively, and achieves the best results on most math benchmarks and all out-of-distribution benchmarks tested.

GRPOQwen3

Qwen3-4B-BaseQwen3-1.7B-BaseAIME

training, research· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper🧠 AI

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

The paper introduces Retrospection-Only Fine-Tuning (ROFT), a minimal online training procedure where an LLM agent generates a self-written explanation of its own attempt at a task and is then fine-tuned with next-token prediction on that explanation alone, without any reward signal or external teacher. Using Qwen3.5-4B on software-engineering tasks, ROFT reaches 49.2% and 26.8% solve rates on held-out SWE-bench Verified and SWE-bench Pro after 20 updates, edging out GRPO's 48.0%/25.3% after 40 updates and making faster early progress, and it can even learn to solve tasks where all 64 sampled base-model attempts initially failed. The authors argue this shows self-generated retrospections act as useful training signal, implicitly performing credit assignment and shaping behavior (e.g., shorter solutions) without RL machinery.

Qwen3.5-4BGRPO

SWE-bench VerifiedSWE-Bench Pro

training, agents· Added 29 Sep 2026, published 28 Sep 2026

huggingface.corepo🧠 AI

MiMo-V2.6-Pro-RL

MiMo-V2.6-Pro-RL is Xiaomi's flagship omnimodal (text/image/video/audio) sparse MoE model, 1.02T total / 42B activated parameters with a 1M-token context, built around scaling RL compute toward self-improvement. It uses a single mixed GRPO run across coding, agentic, visual and cybersecurity tasks, a novel groupwise agentic grading scheme (Groupwise Reward Synthesis + Groupwise Advantage Redistribution) to rank passing rollouts rather than just pass/fail, and a multi-teacher on-policy distillation stage (MOPD2); it also ships a 5-layer speculative MTP decoder and is deployable via vLLM and SGLang.

vLLMSGLangTransformer+2

XiaomiMiMoMiMo-V2.6-Pro-RLXiaomi+6

models, training, multimodal· Added 28 Sep 2026

arxiv.orgpaper🧠 AI

Reinforcing Agentic Creativity in Scientific Ideation with Night Science

This arXiv preprint introduces AI Night-Scientist, an agentic framework that uses GRPO reinforcement learning to counteract LLMs' low-entropy bias toward homogeneous outputs in open-ended scientific ideation. It models creativity along three axes (action, process, outcome) grounded in cognitive science, and trains models to depart from predictable reasoning via semantic guidance rather than just raising decoding temperature. Reported results show a 27.8% expansion in research-direction diversity, a 14.9% increase in contribution types, up to 32.0 points higher predicted citation impact, and 66.2 points higher originality versus the base model.

GRPO

AI Night-Scientist

training, agents· Added 29 Sep 2026, published 28 Sep 2026

huggingface.corepo🧠 AI

MiMo-V2.6-Flash-RL (Xiaomi)

MiMo-V2.6-Flash-RL is Xiaomi's efficiency-balanced checkpoint in the MiMo-V2.6 series, a 309B-total/15B-activated sparse MoE model with native text, image, video and audio support and a 1M-token context window. It's trained with a single mixed reinforcement learning run (GRPO at large batch scale) across coding, agentic, visual and cybersecurity tasks, using a novel Groupwise Agentic Grading scheme (Groupwise Reward Synthesis and Groupwise Advantage Redistribution) to build a self-improvement loop, followed by Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2). The card documents the architecture (MiMo ViT vision encoder, AudioTokenizer audio encoder, 5-layer speculative MTP decoder) and deployment recipes for vLLM and SGLang.

vLLMSGLangTransformer+4

Xiaomi MiMo TeamMiMo-V2.6-Flash-RLMiMo-V2.6+6

models, training, multimodal· Added 26 Sep 2026

arxiv.orgpaper🧠 AI

WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation

WanPE is a 397B-parameter prompt enhancement model trained on 1.05M real-world videos to convert user prompts into director-level cinematic shot plans (camera, lighting, sound, multi-shot structure) for text-to-video generation. It uses video-grounded reverse construction plus Semantic-Consistency GRPO (SC-GRPO) to keep user intent faithful across shots and time, and the authors introduce WanPEval, an 11K-pairwise human-annotated benchmark, showing large human-preference gains (10-50+ points) when powering Wan3.0's video generator versus raw prompts.

WanPEWanPEvalWan3.0+1

research, multimodal, prompting· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper🧠 AI

SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

SpeakerMem-R1 is a dual-track long-term memory architecture for multi-party dialogue agents, storing speaker-labeled verbatim messages alongside derived person-level and group-level state, and combining both at query time by entity, event, and time. The authors train a Writer-R1 model using SpeakerLevenshtein and speaker-conditioned GRPO to reduce attribution/update errors, reporting state-of-the-art results on GroupMemBench, SocialMemBench, and the EverMemBench leaderboard, plus strong performance on LoCoMo.

GRPO

SpeakerMem-R1Writer-R1GroupMemBench+4

agents, research· Added 23 Sep 2026, published 22 Sep 2026

arxiv.orgpaper🛡️ Security

PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

PISmith trains an attack LLM with GRPO in a black-box setting to generate injected prompts that defeat prompt-injection defenses. It addresses extreme reward sparsity, which causes entropy collapse in standard GRPO, with adaptive entropy regularization and dynamic advantage weighting. Evaluated on 13 benchmarks against 8 defenses and 7 baselines (static, search-based such as TAP and PAIR, and RL-Hammer), it achieves the highest attack success rates and also works on InjecAgent and AgentDojo against GPT-4o-mini and GPT-5-nano. It shows state-of-the-art defenses such as Meta-SecAlign remain vulnerable to adaptive attacks.

LLMAgentDojo

PISmithRL-HammerTAP+8

AI Security· Added 17 Sep 2026

huggingface.cowriteup🧠 AI

Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL

A Hugging Face engineering blog post describing how TRL's AsyncGRPOTrainer now syncs only a LoRA adapter (not full weights) to vLLM, letting trainer and inference replicas run as separate Hugging Face Jobs connected via a Storage Bucket instead of NCCL. It details a custom proxy that adds auth headers, routes rollouts to the vLLM replica most likely to hold the matching KV-cache prefix, and broadcasts adapter loads atomically across replicas. Five iterative runs (token-budget batching, disabling gradient checkpointing, raising in-flight caps) cut a 500-step Qwen2.5-Math-1.5B GRPO run from 3h27m to 53 minutes.

Hugging FacevLLMLoRA+4

TRLAsyncGRPOTrainerHugging Face Jobs+10

training, inference, ai-infra· Added 29 Sep 2026, published 10 Sep 2026

huggingface.cowriteup🧠 AI

Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

A Hugging Face blog walkthrough showing how to GRPO-fine-tune LiquidAI's LFM2.5-350M model with TRL and a LoRA adapter (targeting LFM's hybrid attention/convolution modules) using three custom reward functions — JSON/YAML format compliance, field-count matching, and JSON Schema validation — trained on ~500 samples from nvidia's Nemotron-RL-instruction_following-structured_outputs dataset for just 100 steps. On the IFStruct benchmark, the fine-tuned model's overall pass rate rises from 22.6% to 29.7% (JSON specifically from 18.0% to 31.9%), narrowing the gap to the much larger Qwen3.5-2B (33.15%). The post also covers converting the merged model to GGUF and serving/evaluating it locally via llama.cpp's OpenAI-compatible server.

Hugging FaceGRPOLoRA+4

LFM2.5-350MIFStructLiquid4All/ifstruct+4

training, evals· Added 29 Sep 2026, published 3 Sep 2026

huggingface.coarticle🧠 AI

Training a coding model to paint watercolours with TRL and OpenEnv

A Hugging Face engineering blog post reproducing Surya Narreddi's viral "AI paints watercolours" project in the open: a coding LLM (Qwen3.5-35B-A3B, LoRA) is trained with GRPO via TRL to write p5.brush JavaScript sketches that render as watercolour flower paintings, using an OpenEnv RL environment with headless-Chromium rendering, an HPSv3 preference-model reward and a Qwen3-VL-30B-A3B-Instruct pairwise judge scored against a hand-curated 178-image reference pool. It documents the full pipeline (HF Jobs, Spaces, Inference Providers), the LoRA fix needed for MoE models (all-linear targeting), infra bugs found (an OpenEnv websocket caching bug fixed upstream, reward-noise from silent render failures), three reward-mix ablations, and open-sources the dataset, environment, training scripts, rollouts and trained models.

TRLOpenEnvHugging Face+8

Surya NarreddiHPSv3Qwen3.5-35B-A3B+12

training, multimodal· Added 29 Sep 2026, published 3 Sep 2026

huggingface.coarticle🧠 AI

Granite 4.2 LLMs: How They're Built

IBM's technical write-up on Granite 4.2, its first family of dense decoder-only reasoning LLMs (3B, 8B, 30B), post-trained from Granite-4.1 base models pretrained on ~15T tokens with a five-phase schedule extending context to 512K tokens. It details the SFT data mixture (agentic vs. non-agentic, quality filtering with LLM judges GPT-OSS-120B and Gemma 4), a staged asynchronous-GRPO RL pipeline (RLVR, skill boosters, then agentic SWE/Terminal/Search stages for 8B/30B, finishing with RLHF), and the NeMo-RL/NeMo-Gym infrastructure used to run tool-using RL rollouts at scale, plus FP8/NVFP4/MXFP4/GGUF quantized releases and integration guides for vLLM, SGLang, OpenCode, Pi, and OpenHands.

vLLMSGLangllama.cpp+7

Granite 4.2IBMGranite-4.1+21

models, training· Added 29 Sep 2026, published 25 Aug 2026

interconnects.aiarticle🧠 AI

5 useful things you'll learn in my new post-training textbook (shipping now!)

Nathan Lambert (Interconnects) announces his new book, Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs (Manning), which is also freely available online with a full 12-hour video course, code exercises, and model completion comparisons. The post walks through five takeaways: intuitions for RL algorithms (policy-gradient theorem, PPO clipping, GSPO, CISPO), systems tradeoffs in modern RL (asynchronous learner and actor setups, loss aggregation behind DAPO and Dr. GRPO, truncated importance sampling), the three historical eras of RLHF and post-training, a chapter demystifying distillation (including multi-teacher on-policy distillation used by models like Xiaomi MiMo-V2-Flash and DeepSeek V4), and a survey of practical post-training pitfalls such as over-optimization, regularization, evaluation, and character training.

GRPO

Nathan LambertInterconnectsManning+4

training· Added 29 Sep 2026, published 10 Aug 2026

depthfirst.comwriteup🧠 AI

Introducing dfs-large1: A New Frontier for Cybersecurity | depthfirst

dfs-large1 is depthfirst's vulnerability-discovery model, post-trained from GLM 5.2 inside its security agent harness with Fireworks AI. The post describes the multi-task RL recipe: an effort penalty, a finding budget, masked importance-sampling REINFORCE, Dr. GRPO with task-level advantage normalization, MoE router replay and learned context compaction. It reports a shift toward hypothesis testing and executable verification, but gives no benchmark numbers.

GLMGRPOMixture of Experts+1

depthfirstdfs-large1dfs-mini1+13

training, agents, models· Added 29 Oct 2026, published 30 Jul 2026