Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale

What it claims

Beyond the headline results, the paper pins down the models, data sizes, and the fact that there were two separate A/B tests the summary blurs together. Test one: the simulation-guided Card Management (CM) agent (GPT-5.2 backbone) beat the older Card Delivery (CD) agent, lifting tNPS by 36.69 points and SSR by 4.9 percentage points. Test two: using the same simulation recipe, the team screened open-weight models across 16,000+ simulated conversations and 29 variants, selected Qwen3.5-122B-A10B to replace the incumbent CM model, and in a follow-up A/B test it raised SSR by 8.82 pp (highest observed at Nubank) and cut p95 latency by 25% with no significant tNPS change. Simulation is credited with 4.8× faster dev-to-production iteration.

How it works

The simulator is SNOWGLOBE (Guardrails AI), instantiating a persona-driven pattern. An orchestrator reads an agent description and tool definitions (optionally a simulation prompt and historical conversations), plans coverage over M use-case × style slices, generates N personas per slice, and runs K multi-turn conversations each. The key mechanism is tool-boundary simulation: agent tool calls are intercepted and answered with scenario-conditioned synthetic results rather than hitting production backends, with cross-call state consistency, so the agent acts on its real production policy.

Screening is hypothesis-driven: for each candidate change, simulate incumbent vs. candidate, compare evaluator scores against a standing suite plus hypothesis-specific checks; only candidates meeting criteria become eligible for live A/B testing.

Evidence

• CD characterization: 8,000 real conversations split evenly into a development set (simulator config) and test set; 1,000 real conversations per version, 250 synthetic trajectories per version, across 4 CD versions.
• Offline eval uses LLM-as-a-Judge (GPT-4.1-Mini) on five binary categories (card reissue failure, input verification, delivery data check, response conciseness, resolution conciseness), with judge prompts optimized via GEPA.
• Lowest conversation-level cosine distance of 0.035 between real and simulated conversations; version-level evaluator scores show high correlation across the 4 deployed versions.
• Model lineage is explicit: CD versions span GPT-4.1 → GPT-5.1 → GPT-5.2; CM versions all GPT-5.2 with ReAct-style tool definitions, iterated V1–V5.

A naive LLM baseline simulator (gpt-5.6-sol, high reasoning effort) is compared against SNOWGLOBE; it lacks structured coverage orchestration and cross-call consistency.

Limits

The authors are explicit that their diagnostics characterize differences without establishing backend fidelity — they claim only "moderate" conversation-level similarity, and note the general risk that a simulator of unknown fidelity can mislead (score inflation, goal drift). Their central defense is that imperfect simulation can still guide directionally correct improvements, confirmed by live A/B outcomes — not that simulation matches production. All results are on Nubank's Brazilian Portuguese card-support agents; generalization beyond CX card-lifecycle tasks is untested. This is a preprint (dated Sept 2026); the fetched text cuts off mid-Section 3.3, so full offline-evaluation details for CM are not visible here.