Building an AI Text Detector From Scratch

What it claims

Raschka builds a fine-tuned DistilBERT classifier that scores text 0-100 for "AI-generated", in the style of the Pangram models he believes power Substack's AI-detection feature. He then reuses that classifier as a verifier to train a small language model to write text that avoids detection. He frames it as an end-to-end local project covering evaluation, training and local deployment. The stated deliverables are an AI-detector API meant for both humans and agents, plus a user-friendly UI.

He names one practical use beyond spam filtering: use the detector as a guardrail during LLM grammar editing. The example prompt is "Fix my grammar while ensuring that my text still scores 0% AI-generated", which keeps polished human writing from being flagged as AI.

How to use it

The captured text covers only the introduction and method overview. It gives no install commands, dataset, model checkpoint, training cost or API shape. Those details, if the article provides them, come in the implementation sections that were not fetched. The page points to two earlier posts as background:

• his 2023 article on AI-text detection approaches
• "Finetuning Large Language Models", which describes the DistilBERT fine-tuning setup reused here

Limits

The author flags these points, and they matter more than the summary suggests:

• The score is not a probability of AI authorship. It is the classifier's estimated probability for the AI-generated class, relative to its training distribution. He explicitly warns against reading it as a general probability that a text was written by AI.
• False positives are expected. Human-written text will sometimes be flagged as AI-generated.
• Detection is a cat-and-mouse game. A detector learns patterns from current LLMs, and newer models may lack those patterns, by accident or by design. The detector then needs retraining. The evasion half of this project is a demonstration of exactly that failure mode.
• It is described as a small educational project. The goal is to study detector limitations and verifier-based training, not to produce a production-grade detector.
• Results are missing from the fetched text. It reports no accuracy, false-positive rate or evasion success, so the reader cannot yet judge how well either the detector or the evasion model works.