arxiv.orgpaper

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG is a benchmark of 660 evidence-grounded instances (163 real-world gaps from reproducibility reports and GitHub issues, 497 synthetic) that tests whether LLMs can identify and resolve implementation-critical gaps in research-method specifications, i.e. whether a specification gives a coding agent enough information to build the intended method without unsupported assumptions. It evaluates three capabilities across 13 LLMs: codification-readiness assessment, defect localization, and clarification action generation, finding defect localization is the major bottleneck (best model only 9.6% Macro Defect Recovery Rate) while clarification given the defect is much stronger (80.6% success), and an oracle study shows gold resolutions raise codification-ready rate from 14% to 98%.