arxiv.orgpaper

What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity

This arXiv paper systematically evaluates how pruning (depth, width, hybrid, and expert pruning) degrades context-grounded tool calling in smart-home LLM agents, testing four models spanning dense Transformer, dense hybrid, and mixture-of-experts architectures after post-pruning SFT. Using over 19,500 instances from three smart-home datasets, it finds dense models have narrow safe-pruning regions with sharp degradation while MoE models tolerate more pruning, that grounded specificity (device/argument/value) degrades before schema-level intent, and aggressive dense pruning can induce over-refusal. It argues pruned-model selection for tool execution reliability requires evaluation beyond aggregate accuracy.