Towards an Extensible Benchmark for Spoken Dialogue with Social Robots

What it claims

This is a position paper with a benchmark sketch, not a released benchmark. It proposes a co-located collaborative puzzle game as a shared test for human-robot spoken dialogue. The game is designed so that clarification requests, conversational grounding, interruptions and robot initiative arise naturally. It names Retico (https://github.com/retico-team) as the recommended software substrate. The authors frame "robot-ready" spoken dialogue systems against plain LLM interaction. A robot-ready system is speech-driven, works at word level, requests clarification, takes turns fluidly, gives live feedback and handles continuous multimodal input. Plain LLM interaction is text-driven, sentence-level and tokenized, and it assumes understanding. They list five requirements such a system must meet: modular, multimodal, distributive, incremental and time-aligned sensors.

How it works

Base task, shown with a Misty II robot:

• Roles. The robot is the instruction giver and the human is the follower. The roles can be swapped.
• What the robot sees. It sees the target shape, the human's game board, and two puzzle pieces hidden behind left and right dividers. An extra camera may be needed.
• What the human sees. The human sees only the board and the robot's face and arms.
• Pieces. The pieces are abstract pentominoes, shaped like L, T or N. Some share colours, which forces the pair to agree on names for them.
• Distractors. At least two distractor pieces are on the table.
• Hidden-piece rule. The human may take a hidden piece only by swapping another piece in, and only once per divider. A wrong swap makes the puzzle impossible, so the game fails.
• End state. The puzzle is complete and the distractors sit behind the dividers.
• Hardware floor. The robot needs a camera that can be moved to look at the target, dividers and board. Gaze, nods, emotional displays and deictic gestures are optional. Complex manipulation is not required.

Proposed metrics:

• puzzles started and completed
• time to completion
• dialogue turns for each side
• initiations by the human and by the robot
• clarification requests
• interruptions and barge-ins

The authors note that several of these require dialogue-act classification, which the benchmark platform would need to provide.

Proposed extensions:

• Acoustics test suite: white noise, actuator noise, alarms, passing cars, volume changes, and corrupted recognizer output.
• Taboo words: colours, shape names or directions the robot may not say. These can be set at the start or changed mid-game.
• Manipulation and locomotion: the robot fetches the hidden pieces, the roles are swapped, or two robots play together.
• Larger objects: everyday objects, room-scale objects, or objects placed in multiple rooms.
• Gestures: deictic and iconic gestures.
• Player attitudes: adversarial, reluctant, fearful or over-trusting humans, or a robot that is deceptive or lies.

Labs are meant to hold most of the system fixed and vary only the component they study. For example, a lab could use a commercial LLM as a placeholder dialogue module while it studies gestures.

How to use it

Nothing is runnable for the benchmark itself. Retico is the concrete artifact:

• It is free, open source and under active development.
• It has NSF funding, onboarding videos, tutorials and example systems.
• It implements the incremental-unit model of dialogue.
• It supports word-level incremental speech recognition.
• It has camera and microphone input modules and object detection.
• It integrates with Hugging Face language models and agents.
• It has ALFRED benchmark bindings.
• It has robot bindings for Cozmo, Vector, Misty II and Booster K1.
• It supports barge-ins and interruptions.

The paper does not state Retico's licence, install commands or API.

Limits

• No evidence. There is no data, no baseline system, no results, no protocol for human participants and no scoring procedure.
• Tentative design. The authors call the task "one potential benchmark" and invite critique.
• Baseline still to come. They assert that a baseline system with a dialogue task manager should be provided, but have not built it.
• Plans only. Evaluations and community workshops are described as future plans.
• Conflict of interest. Retico is recommended by authors whose prior work extended it, so the endorsement is not independent.

If you need a usable HRI dialogue benchmark today, this does not provide one. If you are building on Retico or want to shape the benchmark's design, it is the reference point.