Skip to content

Back to the catalog

Someone Scraped 5.6 Billion TikTok Videos and Put the Data on Hugging Face for Free
decrypt.coarticle

Someone Scraped 5.6 Billion TikTok Videos and Put the Data on Hugging Face for Free

Decrypt reports that a developer going by "hashfunction," operating under the datasocial account, published a 460GB Parquet dataset of metadata for ~5.6 billion public TikTok videos (captions, hashtags, sound IDs, engagement counts, AI-generated flags) on Hugging Face under CC BY-NC 4.0. The data was obtained by hitting TikTok's private mobile API using generated Android device identities, reverse-engineered request signatures, and a spoofed TLS handshake, bypassing the ToS clause that bars automated scraping; datasocial.ai sells the scraper's source code for $1,699 and offers commercial access, creator profiles, and daily updates. The piece situates this against TikTok's gated official Research Tools and the ongoing Reddit v. Perplexity/SerpApi scraping lawsuit.

TL;DR A dev bypassed TikTok's anti-scraping defenses with spoofed Android device IDs, reverse-engineered API signatures, and TLS spoofing to dump 5.6B video metadata records on Hugging Face — and sells the scraper for $1,699.
Also seen from another angle🧠 AI →
Document

What it is

DataSocial's scraper (sold for $1,699) that produced the 5.6B-video Hugging Face dataset. The page gives the specific evasion mechanism and scale figures the summary doesn't: generated Android device identities, reverse-engineered request signatures, and a spoofed TLS handshake against TikTok's private mobile API, with no login/account required.

How it works

The write-up claims the system collected 3.23 billion creator profiles, 5.94 billion videos, and 2.8 billion comments in three weeks — no authenticated session, just device-identity spoofing plus matching TikTok's internal request-signing scheme and TLS fingerprint to pass as a genuine Android client.

Preconditions / Affected

Applies to anyone pulling data via TikTok's undocumented mobile API rather than its web/public endpoints. TikTok's Section 3.4 ToS bars automated extraction without written approval; this bypasses that via the private API rather than the public site, which the official gated Research Tools (academic/nonprofit, approval-required, limited to US/EEA/UK/Canada/Switzerland) are meant to control.

Caveats

  • Figures (3.23B profiles, 5.94B videos, 2.8B comments, three-week collection time) come from DataSocial's own write-up, unverified by Decrypt.
  • The dataset itself has no creator-handle column in the free Hugging Face release — user ID is only available via DataSocial's paid/commercial tier.
  • A separate, unrelated 4.5-billion-video TikTok dataset is also already circulating on Hugging Face, so this scraping technique/exposure isn't unique to one actor.
  • TikTok is not a party to the Reddit v. Perplexity/SerpApi litigation cited for legal context, and that case concerns scraping via Google search results, not a private mobile API — the legal-precedent angle doesn't directly transfer.

Organisations named here

  • CC BY-NC 4.0
  • datasocial.ai
  • DMCA
  • Reddit

Technologies named here

  • TikTok
  • Hugging Face
  • DataSocial scraper
  • TikTok Research Tools
  • SerpApi
  • Android
  • Perplexity
  • Parquet

Techniques named here

  • Automated web/API scraping
  • Anti-bot circumvention

Threat actors named here

  • hashfunction