Training a coding model to paint watercolours with TRL and OpenEnv

What it claims

The post shows that a mid-size open coding model can be pushed by GRPO toward a hand-curated aesthetic. Three runs on Qwen3.5-35B-A3B with LoRA all improved reward. They differ only in how the weight splits between HPSv3 and the pairwise judge: hps-only, hps-led and judge-led. The page does not state the numeric weights in its text.

The main finding is that HPSv3 alone makes paintings more reliable without making them better. Adding the pool-anchored judge is what raises the top of the distribution and increases pigment use.

How it works

flowchart LR
  P[Policy writes p5.brush sketch] --> G{Gate}
  G -- compile fail / direct p5 / no pigment / text on canvas --> R[Rejected]
  G -- pass --> C[Headless Chromium render]
  C -- timeout or scorer down --> N[None: dropped from group]
  C --> H[HPSv3: is it a flower]
  C --> J[Qwen3-VL judge vs 4 pool refs]
  H --> W[Weighted reward]
  J --> W
  W --> P

• Prompt. The prompt allows only 10 of p5.brush's 47 methods: scaleBrushes, noStroke, fill, noFill, fillBleed, fillTexture, beginShape, vertex, endShape, circle. It gives no API documentation, because a long reference made the model invent methods. The author added one sentence: paint each petal two or three times, big pass first, then a smaller, more opaque one. This noticeably increased colour.
• Reference pool. The 178 pool images are all model-generated. Four open models each wrote sketches from openly licensed iNaturalist hibiscus photos, with three rounds of vision-model critique. The author then hand-rated the results into two tiers, love and okay.
• Judge. Each judge draw takes 2 refs from love and 2 from okay. Every comparison runs in both orders, and the score is the win share. Including the okay tier is a deliberate change from the original, which compares against the top tier only. The point is to give a weak early policy some signal.
• MoE LoRA. The default target_modules only hit 10 of 40 layers on this MoE model. Using all-linear fixes that. The fused routed experts stay frozen even then, but training still worked.
• Early debugging. The first run that learned was a control task with no browser and no judges. It revealed that the learning rate was too low. The fix involved four changes to TRL's GRPOTrainer, which the text references but does not enumerate.

Evidence

All results come from single runs at 110 steps on one subject, a peach hibiscus.

Collapse of bad paintings. Rollouts scoring under 0.3 fell from 99 to 16 across judge-led's thirds, and from 37 to 4 in hps-led. In hps-only, three quarters of the gain came from bad paintings becoming rare. Its best-per-step moved only +0.034, against +0.155 for the median.

Contribution of better good paintings to the group mean.

Paint coverage. Coverage roughly doubled in both judge runs, from 0.11 to 0.23 and from 0.13 to 0.30. hps-only barely moved.

Other findings:

• The judge term itself rose in both judge runs.
• No GRPO group collapsed to identical rewards.
• The model ignores the prompt's "15–30 shapes" instruction, producing 7–9 on average. n_shapes barely correlates with reward (+0.000, −0.14, +0.07 across runs).
• The estimated ceiling for hps-only, if every rollout matched its good ones, is a mean of 0.771.
• Before the fix, infra failures scored 0.0 in about 1.5% of rollouts, rising to 5.2% in the worst run.

Per-metric CSVs are in the repo, and all rollouts are in published datasets.

How to use it

First duplicate the environment Space and the HPSv3 scorer Space. Then set two environment variables that select the reward mix; their names are given in the repo recipe, not on the page. Then launch:

hf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \
  --env-url https://<you>-watercolour-env.hf.space \
  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
  --subject 'a peach hibiscus' --references 4 \
  --top-p 0.95 --top-k 20 \
  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
  --scale-rewards none \
  --steps 110 --n-episodes 240 --num-generations 8 \
  --per-device-batch-size 1 --gradient-accumulation-steps 8 \
  --max-completion-length 8192 \
  --run-tag my-run --out <you>/watercolour-grpo --push-to-hub

Cost.

• A step is 8 rollouts and takes 15–18 minutes, so 110 steps is roughly 27–33 hours of H200 time.
• Rendering takes 70–80% of each step. A single render takes 69–96 s against a 90 s deadline, because the environment Space is CPU-only and renders WEBGL in software.
• The HPSv3 Space must stay up for the whole run and can cost more than training. Pause it or set its sleep timer afterwards.
• The judge runs through Inference Providers.

Swapping in your own pool changes the reward without code changes.

Limits

• No human paintings in the pool. The pool contains only model-made images, which the author calls a real limitation.
• Low diversity. Outputs within a run converge on similar-looking flowers, because the pool covers a single subject.
• Noisy curves. Per-step reward depends on which 4 references were drawn. For example, step 12 scored half a point below step 11 mostly due to a hard draw. GRPO advantages are unaffected, but the curves are misleading.
• Unexplained render slowness. The author has not found the full cause.
• OpenEnv websocket bug. A cached closed socket made every later call fail. If your OpenEnv version predates the upstream fix, check for it.
• Single-turn only. The policy never sees its own render.
• Untested directions:
• multi-step training
• smaller models (a 4B passed the gate in side tests)
• SFT before RL
• judge consistency
• Unverifiable match to the original. The original implementation is unpublished, so choices like all-linear, None-on-failure and stopping at 110 steps cannot be checked against it.