Titan Network supplies fixed-speed connections with unlimited bandwidth through its Web Data Extraction Infrastructure. Teams use these connections for video data collection to train artificial intelligence (AI) models. A connection running at 1 gigabit per second (Gbps) can download up to 324 terabytes (TB) of data in 30 days.
The problem / requirements
A video model learns from video, and the more varied that video is or the more specific the content is, the better the model performs. Teams that build these models therefore collect very large amounts of data from web sources.
There are two major limitations AI companies run into during the collection process. First some video data can only be watched from certain countries, and second many sources apply rate limits, which refuse a visitor that sends too many requests in a short time.
A team facing these limitations needs two things. The first is stable internet protocol (IP) addresses that web sources accept. An IP address shows where a request comes from, like the return address on a letter. The second is connection speed, because video files are large and the speed of the connection sets how much video arrives each day.
Speed is counted in bits and files are counted in bytes, with 8 bits in each byte. A 1 Gbps connection therefore moves 125 megabytes (MB) per second. A gigabyte (GB) is 1,000 MB, a TB is 1,000 GB and a petabyte (PB) is 1,000 TB. One Gbps is 1,000 megabits per second (Mbps). The table shows what 1 Gbps adds up to at full rate.
| Period | Amount per second × seconds in the period | Data volume at 1 Gbps |
|---|---|---|
| 1 second | 125 MB × 1 | 125 MB |
| 1 hour | 125 MB × 3,600 | 450 GB |
| 1 day | 125 MB × 86,400 | 10.8 TB |
| 30 days | 125 MB × 2,592,000 | 324 TB |
These figures are maximums at full speed. To turn speed into video, the examples below use 8 Mbps, a common video bitrate for one full high-definition (1080p) video. At that rate, one hour of video is 3.6 GB.
How the approach solves it
Each connection works as a middleman, called a proxy, between the customer and the website. The customer’s requests go out through IP addresses on real home devices, so the website sees ordinary visitors. The customer picks a fixed speed, such as 1 Gbps, and downloads as much data as that speed allows. The connection is unmetered, so there is no bandwidth limit on the total data.
The IP addresses come from the Proxy Network of Titan Network. Titan Network states that every address comes from a real user device whose owner opted in.
The customer’s engineers write and run the collection programs. Titan Network supplies the connection and the IP addresses. The examples below show the same job for five kinds of team.
Small teams at 0.3 to 1 Gbps
Independent developers and small teams run single connections of 0.3 to 1 Gbps for small-scale video and general web data collection. Their target sources apply geographic and rate limits, so they need highly anonymous egress IP addresses, the addresses their traffic leaves from. At 0.3 Gbps, a connection moves 3.24 TB per day and up to 97.2 TB in 30 days. At 1 Gbps, it moves 10.8 TB per day and up to 324 TB in 30 days, and carries 125 videos at once.
AI startup and data company at 2 Gbps
An AI startup collects first-person videos of humans from public video platforms to pre-train its foundation model, an AI model built to control a physical machine such as a robot. Real-robot data is scarce and expensive, so the model learns from large volumes of internet video of humans. A data company runs a connection of the same size for general web data collection. At 2 Gbps, the connection moves 21.6 TB per day and up to 648 TB in 30 days. It carries 250 videos at once.
Large model teams at 1 to 3 Gbps
The model teams at a search and AI company, an on-device AI company and a consumer technology company collect video from public video platforms and general web data to train their in-house multimodal models. These models learn from multimodal data, which combines more than one kind of input such as text, images and video. The on-device AI team first had Titan Network deliver about 2 PB of finished data in about two weeks, then added its own 1 Gbps connection. At 1 Gbps, a connection moves 10.8 TB per day and up to 324 TB in 30 days. At 3 Gbps, it moves 32.4 TB per day and up to 972 TB in 30 days, and carries 375 videos at once.
Video generation model team at 83 Gbps
Video generation model teams collect video and metadata from public web video platforms to train their models. At 83 Gbps, the connection moves about 900 TB per day and up to 26,892 TB in 30 days, which is 26.9 PB. It carries 10,375 videos at once.
Internet company model team at 150 Gbps
The model team at a large internet company collects public video, audio, subtitles and metadata from video platforms for the pre-training and post-training of its multimodal models. At 150 Gbps, the connection moves 1,620 TB per day, which is 1.62 PB, and up to 48,600 TB in 30 days, which is 48.6 PB. It carries 18,750 videos at once.
Implementation outline
- List the video sources, the countries the traffic must appear to come from, and the dataset size, meaning the terabytes needed.
- Size the connection. Multiply the terabytes needed by 8,000, then divide by the seconds in the period times efficiency. The result is the Gbps required. Efficiency is the share of full rate that a team collects, and it falls below 100 percent because of retries, blocked requests, throttling by the source and idle time. The 8,000 is 1,000 GB times 8 gigabits, and 30 days is 2,592,000 seconds.
- Start with a pilot on a small connection, such as 1 Gbps.
- Check the pilot against success rate, block rate, data completeness and delivery latency.
- Expand the connection once the pilot is accepted.
What good looks like
These are sizing scenarios, not customer results. A team that needs to move 1,000 TB in 30 days needs about:
- 3.1 Gbps at 100 percent efficiency.
- 6.2 Gbps at 50 percent efficiency.
- 12.3 Gbps at 25 percent efficiency.
A larger target is the video dataset that Stability AI described in a 2023 paper, which holds 212 years of video before filtering. At 8 Mbps that is about 6.7 PB. Moving it in 30 days needs about 20.6 Gbps at full rate.
The table shows the 30-day volume in TB of the example connections if the team collects 100, 50 or 25 percent of full rate. Find a connection in the left column and read across.
| Connection | 100 percent | 50 percent | 25 percent |
|---|---|---|---|
| 0.3 Gbps | 97.2 | 48.6 | 24.3 |
| 1 Gbps | 324 | 162 | 81 |
| 2 Gbps | 648 | 324 | 162 |
| 3 Gbps | 972 | 486 | 243 |
| 83 Gbps | 26,892 | 13,446 | 6,723 |
| 150 Gbps | 48,600 | 24,300 | 12,150 |
How Titan supports this
Titan Network supplies the data infrastructure, meaning the connection and the IP addresses, through its Web Data Extraction Infrastructure, and the customer runs the collection.
See Titan Network's Web Data Extraction Infrastructure
Read how Kling AI collects video training data
FAQ
What is video data collection?
Video data collection is the process of gathering video files, along with their audio, metadata and transcripts, from public sources at scale so they can be used to train AI models.
What does unlimited bandwidth mean?
Unlimited bandwidth, also called unmetered bandwidth, means the connection has no cap on total data. The speed in Gbps is fixed, and the volume is not.
How much bandwidth does an AI video dataset need?
Multiply the terabytes needed by 8,000, then divide by the seconds in the period times efficiency. For 1,000 TB in 30 days, the result is about 3.1 Gbps at 100 percent efficiency and 6.2 Gbps at 50 percent.
What is the difference between Gbps and Mbps?
Both measure connection speed. One Gbps is 1,000 Mbps, and 1 Mbps moves 0.125 MB per second.
How many GB is one hour of video at 8 Mbps?
One hour of video at 8 Mbps is 3.6 GB. To convert a video bitrate in Mbps to GB per hour, multiply by 0.45.
What is multimodal data?
Multimodal data is training data that combines more than one type of input, such as video, audio and text.
Do the IP addresses come from real user devices?
Titan Network states that every IP address comes from a real user device whose owner opted in through the Titan app and earns rewards for sharing the connection. The Proxy Infrastructure page explains how it works.
