SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

What it claims

The paper reports results, not just a benchmark design. Across 3,600 baseline sessions (EXP-1, four models), the average conversion rate (CR) is 44.4%, the average Buyer Resistance Score (BRS) is 7.11/10, and the average role inversion rate (RIR) is 0.53%. Of the four generator models, Phi3.5-mini converts most (70.7% CR), but Llama3.1-8B is more stable, and Llama3.1-8B is the model chosen for the controlled experiments. Shortening conversations has the largest effect on outcomes: a 5-turn cap drops CR to 16.1%. The authors also report that estimates stabilize within 148–228 sessions per model, so about 230 sessions per model is enough for replication.

How it works

Each session is a LangGraph StateGraph. Both agent and buyer turns are RAG-grounded. On the buyer side, each persona has four objection facts (48 in total), and the top 2 are retrieved by keyword overlap with the agent's last message.

flowchart TD
  P[PhonePitch: budget→tier] --> A[AgentSpeak]
  A --> C[CustomerSpeak + top-2 RAG facts]
  C --> R[AgentResponse]
  R --> S[ScoreTurn]
  S --> E{EXP-2: turn≥5 and 3-turn intent ≤4.0?}
  E -- yes, once --> M[Swap to Rajesh, Rs.5,000 authority]
  M --> T
  E -- no --> T{Purchase, hard-no, or max turns?}
  T -- continue --> A
  T -- stop --> J[3 Ollama judges + rule-based CR]

• Agents. Each of the five agents has a discount cap between Rs.1,000 and Rs.5,000. Agents follow a phased prompt schedule: discovery → pitch → objection handling → soft close → final offer.
• Buyers. Each of the twelve personas has a hard budget (Rs.15,000–1,20,000), a core fear, a specific "persuasion arc" trigger, and English/Hinglish soft-yes and hard-no phrases. A hard-no ends the session early.
• Purchase detection is rule-based, not judged. A session counts as a purchase if either of these happens:
• one of 18 purchase phrases appears (e.g. "okay send the link", "haan bhejna"), or
• the buyer gives a positive affirmation within two turns of the agent mentioning a payment link.

Intent-score thresholds are excluded on purpose, to avoid inflating CR.

• BRS is the mean of 1–10 judge scores over five sub-dimensions: price, specification, emotional, character and objection. The paper treats BRS ≥ 6.5 as a realistic challenge and BRS < 4.0 as persona breakdown.
• SMS is a weighted mix of four components: 0.35 × exponential resistance-decay fit, 0.25 × elasticity to tactic switches, 0.20 × tipping point (first turn with intent > 6.0), and 0.20 × BRS.
• Judges are three small local models: Llama 3.2:3B, Qwen2.5:7B and Mistral:7B.

Evidence

• Across models, BRS ranges from 6.70 to 7.47 and fidelity from 6.95 to 8.17.
• Gemma2-2B has a collapsed Persona Fidelity Score (PFS 3.97), which the authors tie to volatile emotional trajectory.
• Human validation used 50 internal sessions: κ = 0.71, and Pearson r = 0.81 with BRS.
• SMS weights were chosen because they correlate r = 0.81 with human judgments, versus 0.74 for equal weights and 0.69 for outcome-heavy weights.
• Against SalesLLM, the closest prior work, the paper claims RIR under 8% without specialised training. SalesLLM reduced RIR from 17.44% to 8.8% using an SFT+DPO CustomerLM.

How to use it

The paper says persona phrase lists are "provided in the open-source release," but the fetched text gives no URL, licence or run command. The setup runs locally, with Ollama judges and sub-10B generators.

Domain specifics may limit transfer:

• The product catalogue is 15 Samsung phones priced Rs.17,999–1,64,999, read from a single JSON file.
• Personas use Hinglish.
• Objection patterns come from Indian retail.

Limits

• Unverified numbers. Phi3.5-mini's PFS and ETI outputs are excluded pending transcript verification. Its leading CR comes from the same runs, so treat that figure with care.
• No RAG ablation. All reported results use bilateral RAG, and the ablation is only "planned," so you cannot tell how much grounding contributes.
• Exploratory metrics not reported. ITS, ARI and SPI are logged but not reported.
• Weak validation. The judges are 3B–7B local models, and human validation is internal with n = 50. The SMS weights were tuned on that same 50-session set, so the r = 0.81 is in-sample.
• Internal inconsistencies:
• Related work refers to a "three-model comparison," while elsewhere the paper says four models.
• The fetched text does not name the fourth model.
• The "<8% RIR" claim does not match the 0.53% reported in the abstract.
• Truncated source. The fetched text cuts off partway through the evaluation protocol, so later results and stated limitations were not available.