90% of US working hours are spent on tasks not covered well by AI benchmarks

What it claims

Of 18,797 O*NET tasks across 923 occupations in 22 job families, few are tested by current AI benchmarks. Only 208 occupations have a task matched by at least 5 prompts from the 29 selected benchmarks. Six of the 22 job families have no such occupation at all. Weighting by time spent puts about 90% of US working hours on tasks with little or no coverage.

Coverage is skewed in three measurable ways:

• AI exposure. Tasks rated "exposed" by Eloundou et al. are about 6x more likely to appear on benchmarks than "unexposed" tasks.
• Wage bill. Tasks from occupations with larger wage bills are overrepresented even after controlling for exposure.
• Task importance. After controlling for occupation and exposure, tasks at the top of O*NET's 1–5 importance scale are about 6–10x more likely to be covered than those at the bottom.

The concrete example given: lawyers' tasks are heavily covered, but the similarly AI-exposed tasks of legal secretaries and court clerks barely are.

How it works

1. Select 29 benchmarks for quality and economic relevance, favouring unsaturated, public, agentic ones. The number per job family is capped to maximise breadth. The full list and selection criteria are in a separate Benchmark Index tab that was not fetched.
2. An LLM judge tags each prompt with as many O*NET tasks as it judges relevant, following Wang et al. One prompt can map to several tasks.
3. The tagging is checked against human expert tags on Mercor's APEX benchmarks, a subset of the 29. The page reports no agreement figure.
4. A task counts as "covered" at 5 or more mapped prompts.
5. Covered tasks are combined with Hatgis-Kessell et al.'s estimates of each task's share of working time. Those estimates are themselves LLM-derived. The result is the hours-weighted 90% figure.

Evidence

The page tested ONET-weighted occupation scores on APEX for investment banking, law, accounting and software engineering. Each ONET task gets equal weight, compared against a simple average over occupation-relevant prompts. The two scores were "generally quite similar", because model performance was similar across the included tasks. Fewer than one third of the relevant O*NET tasks for these four occupations had enough APEX coverage to be included. The page does not give per-occupation numbers or model names in the text.

Limits

• Provisional, and biased low. Any incomplete benchmark list understates coverage, and the authors say the size of that bias is hard to bound. The per-family cap also understates coverage in heavily benchmarked families such as software engineering.
• Unsaturated-only selection. This may understate coverage of tasks that are easy for LLMs. The authors judge this a smaller bias than including saturated benchmarks, which may be less realistic.
• Subjective criteria. Selection criteria such as "Economic Realism" involve judgment.
• Two LLM-estimated inputs in the headline. Both the task tagging and the time shares are LLM-derived, and no validation metric for the tagging is published.
• Thin data per task. Few items per task leave the estimates underpowered, and one or two bad items can skew a task's score.
• No quality weighting. Some items are simplistically graded, contain errors, or can be reward-hacked, and nothing currently down-weights them.
• Automation only. The method measures end-to-end automation, not how much a model augments a human. The authors call their augmentation work early and say they have no scalable method yet.
• Physical work excluded. Citing Dingel and Neiman, only 37% of US jobs could be done fully from home, so about 63% involve physical tasks that current benchmarks cannot easily capture.

For an engineer: treat the per-occupation shading as a lower bound. Do not use it as an occupation-level capability score yet. The page itself shows that the weighted scores rest on less than a third of the relevant tasks.