en.cryptonomist.charticleGPT-4o scored 0.82 on non-public books—what does it mean for OpenAI’s GPT training data?
Beyond Public Access in LLM Pre-Training Data is a study by Ilan Strauss that applied the DE-COP membership inference attack to 34 O'Reilly Media books (13,962 paragraphs). GPT-4o scored an AUROC of 0.82 on non-public books, against 0.54 for GPT-3.5 Turbo and 0.56 for GPT-4o Mini, where 0.50 is chance. The author notes a small book sample and possible control-data contamination as limits.
TL;DR Reports a DE-COP membership inference study suggesting GPT-4o recognises paywalled O'Reilly books that smaller OpenAI models do not.
Organisations named here
- OpenAI
- Beyond Public Access in LLM Pre-Training Data
- Ilan Strauss
- O'Reilly Media
Technologies named here
- GPT-4o
- GPT-4o Mini
- LLM
- GPT-3.5 Turbo
Techniques named here
- DE-COP
Concepts named here
- AUROC
- Control-data contamination
- Pre-training data provenance
Related in research
