Guide to Scraping TikTok's Private Android API Published, 4.5B Videos Dataset Released

Accessing TikTok's Private API

A new technical guide outlines the process for scraping data directly from TikTok's private Android API. This API is distinct from the public web endpoints, offering significantly faster data access and a broader range of information. The guide covers critical aspects such as device registration, request signing, regional host partitioning, and TLS handshake fingerprinting.

Challenges of API Interaction

Interacting with this private API presents specific challenges. Four elements must be correctly configured simultaneously: a TikTok-issued device credential, a valid request signature, the correct regional host, and a TLS handshake that mimics a mobile device. Failure in any of these results in an HTTP 200 response with an empty body, providing no error feedback to indicate the source of the problem.

Data Collection and Dataset Release

Using the methods described, a system collected 3.23 billion creator profiles, 5.94 billion videos, and 2.8 billion comments within three weeks. A subset of this data, specifically 4.5 billion video records, has been uploaded to Hugging Face as a free dataset. This dataset includes video captions, view counts, like counts, comment counts, save counts, associated sounds, country of origin, and posting times.

Available Data Points

The API allows for the extraction of various data points, including creator profiles, all videos posted by a creator, follower and following lists, TikTok's similar-creator graph, and full video details with complete statistics. It also enables access to comments and replies, sound information, trending sounds, hashtags, and keyword searches across videos, creators, and sounds. The guide documents 24 distinct endpoints, each with measured success rates.