Show HN: Training a model to identify AI web content from structure alone

What it claims

SlopShape detects AI-generated B2B blog posts from structure alone — how a post organizes information, orders points, cites evidence, and sets voice — after explicitly stripping out word-level style. It is a domain transfer of StoryScope (which did the same for fiction) to commercial writing, and the summary's headline numbers come with specifics worth knowing: the 214-feature instrument yields 187 purely structural features, the classifier is XGBoost with SHAP for the feature-importance analysis, and ten features carry most of the signal. The five source models are named: gpt-5.4, claude-sonnet-4.6, gemini-3-flash, deepseek-v3.2, kimi-k2.5.

How it works

Human posts (2,250, from pre-ChatGPT Wayback snapshots 2008–2022) are each reverse-engineered into a content brief; each brief is given to all five models, producing five mirror posts per human original — a paired corpus. An LLM (gpt-5.6-terra) first derives an 11-dimension commercial schema bottom-up from human posts only (purpose, audience, structure/flow, explanation, evidence, voices, actionability, commercial integration, timeliness, page format, writing style), then extracts a JSON template per post, discovers features, and scores. A dedicated Writing-Style dimension exists precisely so it can be excluded — that exclusion is what lets the paper attribute detection to structure. Splits are domain-disjoint (198/32/38 train/val/test) with the feature-discovery set (100 prompts) held out from all classifier splits.

Evidence

• 98.0 macro-F1 on held-out companies; 98.1 when every AI post is reworded by its own model.
• The central structural finding: human posts occupy rare structural configurations at mean rarity 0.84 vs pooled AI 0.44, Cohen's d = 1.83. Per-model AI rarity ranges 0.33 (GPT-5.4) to 0.55 (DeepSeek).
• Attribution 79.3% across six sources vs 16.7% chance.
• Controls: a length-only classifier cannot separate the classes; a 13-gram contamination check flagged only 0.19% of pairs (excluding them → 97.9). Human–human annotation kappa 0.928, human–model 0.946.
• Context for the motivation: word-level detectors hit near-perfect on unedited text but collapse under paraphrase (DetectGPT 70.3%→4.6% at 1% FPR, per Krishna et al.).

Limits

• Scope is narrow by the authors' own statement: single-pass posts from these five specific models, as generated or reworded. No claim about multi-model pipelines, heavy human editing, or other domains.
• Corpus skews recent (75.5% of snapshots 2020–2022) and toward software/SaaS (26.5%); B2B blogs only.
• Release is partial: pipeline, instrument, prompts, code, and aggregate artifacts are public, but post-level data is available to researchers on request only — you cannot fully reproduce from the public drop alone.