Video Data Is Still in Demand: 6 Datasets, Led by a 5.6B-Record TikTok Release   

Search interest in datasets for AI remains strong: roughly 1.7 million searches in the past month alone. Video data - especially short-form social video with engagement signals - is one of the most active corners of that demand.

The info above is from Google Trends plus extension:

• https://trends.google.com/trends/explore?date=today 5-y&q=data for AI&hl=en-US
• meetglimpse

Below are six new big video datasets worth knowing, led by a 5.6 billion-record TikTok metadata release, followed by five close alternatives for recommendation, multimodal work, and creator-dynamics research.

The six datasets at a glance

1. datasocial/tiktok-5.6B-videos - the headline release

This is the dataset that anchors the current wave. It holds metadata on roughly 5.6 billion public TikTok posts, spanning fields such as

video_id, posted_at, country, language, caption, hashtags, sound_id, on_screen_text, is_ad, is_ai_generated, and full engagement counts (views, likes, comments, shares, saves, downloads, engagement_rate).

The licence is CC-BY-NC-4.0, and the size category is 1B–10B records. Because of its scale, it is best queried with DuckDB, Polars, or streaming rather than loaded fully into memory. For anyone studying short-video virality, hashtag propagation, or sound trends at population scale, this is currently the reference point.

Viewer link sorted by views(which doesn't work for me): datasocial/tiktok-5.6B-videos viewer

To read the top records or without downloading the whole dataset check the bottom of the article.

2. kuben-developer/tiktok-videos-4b - the near-twin

At roughly 4.5 billion TikTok video records, this set covers much of the same metadata ground: captions, engagement counts, music/sound IDs, duration, country/language, and ad flags, collected through TikTok’s mobile API. It is slightly smaller than the 5.6B release but remains one of the largest public TikTok metadata dumps, making it a practical mirror or complement when you need cross-validation or want to work with a marginally lighter footprint.

P.S. the original link now gives 404 - https://huggingface.co/datasets/kuben-developer/tiktok-videos-4b

3. deepvk/VK-LSVD - recommendation-scale interactions

This is a different kind of video dataset. VK-LSVD contains:

• 40+ billion user–item interactions from VK
• covering about 10 million users
• 20 million short videos over six months.

It includes rich feedback signals - watch time, likes, dislikes, shares, bookmarks, comment opens, author clicks - plus content embeddings and context. If your work is about ranking, sequential recommendation, or industrial recsys rather than pure video metadata, this is the strongest complement on the list.

VK-LSVD is the largest open industrial short-video recommendation dataset with real-world interactions:

40B unique user–item interactions with rich feedback (timespent, like, dislike, share, bookmark, click_on_author, open_comments) and context (place, platform, agent);
10M users (with age, gender, geo);
20M short videos (with duration, author_id, content embedding);
Global Temporal Ordering across six consecutive months of user interactions.

4. The-data-company/TikTok-10M - smaller but feature-rich

10 million TikTok posts may sound modest next to the billion-scale sets, but this dataset trades raw volume for richer content features: descriptions, engagement, music, POI, and quality scores.

That makes it well suited to multimodal and content-understanding experiments where per-item feature depth matters more than having every video on the platform.

5. lingbow/tiktok-video-engagement-1m - the longitudinal panel

Most large video dumps are snapshots. lingbow’s collection is structured as panel data: video metadata, 30-day engagement trajectories, creator daily stats, and trending hashtags/music for roughly 1 million videos.

This design makes it unusually good for studying creator dynamics, trend adoption over time, and temporal engagement patterns - questions that are awkward to answer with a single flat metadata dump.

6. Rijgersberg/YouTube-Commons - longer-form video-language data

The outlier on the list is YouTube-Commons, which provides 22M+ transcripts from millions of CC-licensed YouTube videos across many channels. It is not short-form TikTok data, but it carries the same “massive public video metadata + text” spirit for longer-form video-language and multimodal work. If your pipeline needs aligned speech/text at scale, this is a natural addition.

Summary

All are freely available on Hugging Face, and the multi-billion-record sets reward streaming-first tooling over in-memory loading.

The 5.6B data set was released with Scraping TikTok's Mobile API guide which has been taken down at the request of TikTok. Few of those datasets were also removed or moved.

Scraping TikTok's Mobile API has been taken down at the request of TikTok. The guide to TikTok's private mobile API is no longer publicly available.

Bonus: Partial downloading - stream rows instead of pulling the whole dataset

Since the dateset is big we can stream the rows to read to top videos order by views by:

import pandas as pd
import requests
from io import BytesIO

url = (
    "https://huggingface.co/datasets/"
    "datasocial/tiktok-5.6B-videos/resolve/main/"
    "data/2018-06.parquet"
)

response = requests.get(url, timeout=180)
response.raise_for_status()

df = pd.read_parquet(BytesIO(response.content))

top100 = (
    df.nlargest(100, "views")[
        ["video_id", "views", "likes", "caption", "posted_at"]
    ]
)

print(top100.to_string(index=False))

the result is the most watched videos from this data set:

          video_id    views   likes                                                                                             caption                 posted_at
6567254299959627013 41136475 3513978  Where did the stair go? Let me know 🌎 #shuffle #stairshuffle #stair #featureme IG: antonielokhorst 2018-06-15 10:20:46+00:00
6564064216309828870 30692807 1948967                                         Les presento a mi curso de danza💙 #argentina #handchallenge 2018-06-06 20:01:37+00:00
6572151285066239238 29354763  665279                                                                                                     2018-06-28 15:03:34+00:00
6571026293507230981 23841866 1530695                                                                          #watermelone @Logeshantony 2018-06-25 14:18:01+00:00
6572213220486941958 23339846 2608218                                                                 this was my song in 4th grade aLmao 2018-06-28 19:03:54+00:00
6569875521021152517 21332903  647149                                                                                    #viratkohli 🙌🏻🙌🏻 2018-06-22 11:52:26+00:00
6564013123936718086 21098132 2135118                                                                       hi babydolls I LOVE U SO MUCH 2018-06-06 16:43:21+00:00
6572142286396722438 20410840 1061082                                   Lol I can’t dance for nothing but #spammylikebuttonplease 😒😒❤️🌸😂😂 2018-06-28 14:28:39+00:00
6572634489799838981 20193082 1892049             https://huggingface.co/docs/datasets/stream                                                                             OH MA GAWD 2018-06-29 22:18:39+00:00
6569540344470310149 16298642 1385448   #shuffletutorial This is how to do the “party machine”👯♀️ #shuffle #shuffledance #dance #tutorial 2018-06-21 14:11:47+00:00
6562361557408288001 15402486  559495                                                                                                 め！！ 2018-06-02 05:54:37+00:00
6572985214589472005 15255591 1831981        I bedazzled my face😱😂#BeautyconAuditionLa #foryoupage #comedy #URAdorable #highlightyourlife 2018-06-30 20:59:38+00:00

The tiktok video format is:
https://www.tiktok.com/@july_balla/video/6564064216309828870

The code retrieves a single monthly Parquet file over HTTP and loads it into a Pandas DataFrame in memory using BytesIO, avoiding the need to save the file to disk. However, response.content buffers the entire downloaded file in RAM, and pd.read_parquet() materializes its data for processing.

Given the dataset's 5.6 billion rows and 460 GB total size, downloading and processing all files this way would be impractical. Instead, use streaming or chunked processing with DuckDB, which can scan Parquet files remotely, push down column projections and filters, and perform aggregations without loading the entire dataset into memory.

To find the global top 100, each file can be scanned incrementally, retaining local top candidates and merging them into a final ranking, reducing memory requirements while still requiring substantial I/O across the dataset.

Alternative approaches for streaming the dataset

DuckDB failed to query the dataset directly, and the Hugging Face dataset viewer API returned a server-side authentication error. The Stack Overflow approach suggests streaming with load_dataset(..., streaming=True) to process records incrementally without downloading the entire dataset.

However, with 5.6 billion rows and 460 GB of data, streaming alone does not guarantee finding the global top 100 efficiently: it may still require scanning every record.

• https://pypi.org/project/duckdb/
• https://stackoverflow.com/questions/75481137/is-there-is-a-way-that-i-can-download-only-a-part-of-the-dataset-from-huggingfac
  • https://huggingface.co/docs/datasets/stream