Kling AI, the video generation model of Kuaishou Technology, runs video data collection through Titan Network web data extraction infrastructure at 83 gigabits per second (Gbps). That is about 900 terabytes (TB) of data per day, 26.9 petabytes (PB) a month and about 327 PB a year.
The company
Kling AI is the video generation model of Kuaishou Technology. It turns text, images and references into video and images for filmmakers, creative studios, advertising and marketing teams, e-commerce brands, music producers and developers who build on its application programming interface (API). Its tagline, āYou call the shots,ā describes the goal: a creator describes a scene, and the model produces it.
A video generation model is an artificial intelligence (AI) system that creates video. It learns from examples, so what it can make depends on how much video it was trained on and how varied that video is. This kind of training material, which combines video with audio, text and metadata, is called multimodal data, and video datasets are a central part of it. Every improvement Kling AI ships to creators, from more realistic motion to sharper detail, depends on the model learning from more real video.
The challenge
Video generation models are very sensitive to the scale and diversity of their training data. To keep improving, Kling AI collects video from the public web continuously and in very large amounts.
āOur models get better with every hour of real video they learn from, so the speed of collection sets the pace of improvement.ā
Kling AI engineering team
Two things make this hard. First, limited access. Some videos can be watched only from certain countries, and other sources apply rate limits, which means they refuse a visitor that sends too many requests in a short time. Limits like these are a standard obstacle in video scraping at scale. Kling AI therefore needs a very large pool of stable internet protocol (IP) addresses. An IP address is the internet address a request comes from, and a residential IP address is the address of a real home connection. Second, video files are large, so the speed of the connection decides how much video can be collected each day.
The scale is easier to see in hours. One hour of 1080p video at 8 megabits per second (Mbps) is roughly 3.6 gigabytes (GB). At about 327 PB a year, Kling AI needs room for about 90 million hours of video a year, which is more than 10,000 years of nonstop playback.
The solution
Titan Network supplies Kling AI with a fixed-speed connection of 83 Gbps with unlimited bandwidth through its web data extraction infrastructure. The connection works as a proxy, a server that sends a customerās requests to a website from a different IP address. The fixed speed gives Kling AI predictable collection capacity. At 83 Gbps, the connection can move roughly 900 TB of data a day, with no cap on the monthly total. Kling AI went from its first test to the full 83 Gbps within one week.
The connection draws on a pool of more than 40 million IP addresses, so Kling AI can reach video sources in different locations and collect at high request volumes. Titan Network states that every address comes from a real user device whose owner opted in and earns rewards for sharing the connection.
Getting the connection in place was only part of the work. Every collection job behaves differently, so Titan Network worked with Kling AI to tune the setup as traffic increased. Within a few days, the success rate reached 99.8 percent, so nearly every request returned the data Kling AI expected.
Kling AIās engineers continue to run their own collection workflows. Titan Network provides the access and network capacity that keep those workflows running at scale.
The results
- 83 Gbps of fixed capacity with no limit on total data, running at full speed within one week.
- About 900 TB of video data per day, 26.9 PB a month and about 327 PB a year.
- About 250,000 hours of 1080p video per day, about 7.5 million hours a month and about 90 million hours a year.
- A 99.8 percent success rate, reached within a few days.
āTitan Network had 83 Gbps running for us within a week and tuned it to a 99.8 percent success rate in a few days. Our engineers now spend their time on the models, not on the connection.ā
Kling AI engineering team
For scale, in a 2023 paper, Stability AI described a video training set of 212 years of video, before filtering. At 8 Mbps per video, that is about 6.7 PB, which Titan Networkās web data extraction infrastructure carries for Kling AI in about 7.5 days.
Frequently asked questions
What is video data collection?
It is the process of gathering video files, along with their audio, metadata and transcripts, from public sources at scale so they can be used to train AI models. Kling AIās video data collection runs at 83 Gbps, about 900 TB a day.
How much video data does an AI video model need?
It depends on the model, but scale and variety both matter. Kling AI collects about 327 PB of video a year, roughly 90 million hours of 1080p video.
What is a rate limit in web data collection?
A rate limit is a cap on how many requests a site accepts from one address in a short time. At this scale, requests are spread across a very large pool of IP addresses, more than 40 million in Kling AIās case, so no single address reaches the limit.
What is multimodal data?
Multimodal data is training data that combines more than one type of information, such as video, audio and text. Video datasets are a central part of the multimodal data used to train AI video models.
