The gap in shared understanding of LLM capability is widening

What it claims

This is an opinion thread, not a study. Karpathy says the gap between what different groups have seen LLMs do is widening, and that it now forms a four-tier funnel. The summary gives the tiers. The thread adds three things the summary leaves out:

• Human oversight is lagging at the top tier. Karpathy writes that "human review and comprehension are starting to fall behind." The examples given are people still doing "archeology" on the OpenAI-HF incident months later, and mathematicians who may spend a long time working through "722 manuscripts on frontier mathematics."
• Recency matters. The thread's continuation says many people tried free-tier ChatGPT "somewhere last year" and let that shape their view of AI too much. An old, free-tier impression is a poor guide to current frontier capability.
• The tier-3 shift is recent. Under a year ago, Karpathy says, they were writing code by hand with occasional Tab-autocomplete. Now agents complete projects that used to take weeks or months: building, copying, translating and decompiling apps from binaries.

How it works

Each of the three causes drives a different dimension of the funnel:

1. Ambition, problem size and horizon drive the consumer vs. professional split. "A question with a paragraph answer barely stresses the system." You only see the capability if you have "a reservoir of big, difficult problems that you really care about."
2. Jaggedness drives the split by domain (math and code vs. everything else). The mechanism is narrower than "RL on verifiable rewards." Karpathy says training uses "a curated environment mixture driven by revenue potential." That is why capability peaks where tasks are digital, verifiable and economically valuable.
3. Access (free-tier, paid-tier or internal) drives the visibility split.

For an engineer judging a tool, points 1 and 3 suggest a practical test. Assess it on a large, long-horizon task you care about, using a current paid or frontier tier. Short queries on a free tier will understate it.

Limits

• The population figures (~6B, 1–2B, ~20M, ~5,000) are self-described "napkin math." No source is given.
• The top-tier picture comes from what "the external world has seen [as] the preview," not from direct evidence presented in the thread. It covers swarms of thousands of agents working for weeks, zero-days, machine-speed cyber offense and defense, and new science and mathematics.
• Karpathy notes that execs and other professionals outside math and code understand the shift only "second-hand." The claim of modest uplift for most professionals is an impression, not something measured.
• The fetched text cuts off partway through the recency argument, so any further points there are missing.