arxiv.orgpaperIn-Context Robot Learning with VLM Agents
This paper introduces GPT-Policy, a framework for in-context robot learning that uses a vision-language model to propose robot-tool actions from demonstrations and interaction feedback without gradient updates. It combines a context compiler that preserves task-relevant visual transitions with a constrained controller that verifies and executes each proposed action and reports outcomes. Real-robot trials show human video demonstrations improve task completion even without robot action labels, and aligned action references further help on contact-sensitive tasks.
TL;DR A VLM-agent framework (GPT-Policy) that learns robot tasks in-context from demonstrations, no fine-tuning required.
Technologies named here
- GPT-6 Astra
- GPT-Policy
Techniques named here
- In-context learning for robot policies
- Context compiler
- Constrained controller
Related in research