Your scraper returns a 403. You check the obvious things first. The IP is residential, not datacenter, so that’s not it. It’s freshly rotated, never used on this target before, so it’s not burned. You check the headers against a real browser’s, and they match. You check the cookies, and they’re clean. You even slow down the request rate, just in case. Same result. Every check you know how to run says this request should have worked.
It didn’t. And the reason has nothing to do with anything on that list. Something about how the connection itself opened gave you away, before your request even reached the site. Not the IP. Not the headers. Something earlier than both of those, happening before your code even gets a say.
This is the part most scraping setups get wrong, and it’s costing teams more time than they realize. Bots now make up 53% of all internet traffic, and 40% of that is outright malicious, according to the 2026 Thales Bad Bot Report, the seventh year in a row that number has grown. Every site worth scraping has gotten better at spotting the difference between you and a real visitor. Rotating IPs and hoping for the best stopped being a real strategy a while ago.
Diagnose why your scraper keeps failing. Contact us for a scraping infrastructure review
Why the address alone doesn’t fool anyone anymore
A few years ago, this problem had a simple fix. Send too many requests from one IP address, and that address stands out fast, the same way one visitor ringing your doorbell a hundred times a minute would stand out. So the fix was obvious: spread your requests across a lot of different addresses instead. No single one ever looks like it’s flooding the site. For years, that alone was enough, because the systems watching for this were mostly just counting requests per address.
That stopped working once those systems started looking at more than the address. Here’s the problem: if your script rotates through a hundred IPs, but every single request opens its connection the exact same way, same software, same settings, every time, you haven’t actually become a hundred different visitors. You’ve become one visitor wearing a hundred different name tags. And that repeating pattern is a much stronger tell than a busy IP ever was. A hundred real people, on a hundred real devices, don’t connect identically. A script does, unless something is built specifically to stop it.
There’s a technical name for this: a TLS fingerprint. The mechanics get complicated fast, so here’s the plain version. Every browser and every scraping tool has its own way of saying “hello” the instant it opens a connection, before a single word of your actual request goes out, and that hello is detailed enough to work as an identifier on its own. This isn’t theoretical. When Chrome started deliberately randomizing part of that hello in 2023 specifically to make fingerprinting harder, Fastly’s own network data showed the share of Chrome traffic matching the previous common fingerprint drop to near zero within days, which tells you how precisely identifiable that “hello” was in the first place. A real browser’s version of it looks like millions of other real browsers’. A script running default settings says hello in a way no person’s device ever would, and a detection system that’s seen that exact hello before can flag it instantly, without ever checking your IP.
This is also why simply switching proxy providers rarely fixes anything on its own. Most of what providers compete on and advertise, more IPs, more countries, higher success rates on paper, lives entirely on the address side of the connection. The hello happens somewhere else entirely, a layer most proxy shopping never touches.
A quick reference for the errors you’ll actually see
Cloudflare sits in front of roughly 20% of all websites worldwide as of late 2025, and it’s close to universal among the sites worth scraping at scale. So if you’re staring at a failed request, there’s a good chance you’re looking at one of a handful of specific error codes. They don’t all mean the same thing, and treating them like they do is how you waste an afternoon fixing the wrong problem. Here’s what each one is actually telling you, including the 403 that opened this article.
- 403 Forbidden. This is the generic “access denied” response, and it’s frustrating precisely because it’s generic. Unlike Cloudflare’s specific numbered codes, a 403 doesn’t explain itself. It’s what gets served the moment a system decides not to let a request through, whatever the underlying reason, a suspicious IP, a suspicious fingerprint, or something else entirely, all look identical from the outside. That’s exactly why checking the IP, the headers, and the cookies turned up nothing at the start of this article: a 403 caused by a fingerprint mismatch produces the exact same response as a 403 caused by anything else. The error itself never tells you which layer failed. Working that out is the actual debugging.
- Error 1015, “you are being rate limited.” This one genuinely is about detection, specifically about behavior rather than identity. Cloudflare’s rate-limiting rules trigger 1015 when a visitor, human or automated, sends more requests than the site owner’s configured threshold allows, or when the overall pattern reads as automated regardless of speed. If you’re seeing 1015, the fix that actually works is slowing down and keeping each session’s signature consistent from request to request, not disguising it differently every time.
- Error 520. This one usually isn’t about detection at all. Cloudflare’s documentation defines it as the origin server returning an empty, unknown, or unexpected response, commonly caused by server crashes, misconfigurations, or headers that exceed Cloudflare’s size limit. The most common cause when it’s happening to a scraper specifically is your own request rate overwhelming a server that wasn’t built to handle that much traffic. The fix is sending fewer simultaneous connections against that target, not a better disguise. A better fingerprint doesn’t help you here, because a fingerprint was never the problem.
- Errors 521 and 522. Both are almost always the target’s server having a bad day rather than anything about your scraper. Cloudflare documents 521 as the origin refusing the connection outright, often because the server is down or its own firewall is blocking Cloudflare’s IP ranges by mistake. A 522 means the connection attempt timed out completely before a handshake could even complete, typically because the origin is too overloaded to respond in time. Neither is something you fix on your end beyond confirming the site is actually reachable right now.
- "Your IP has been temporarily blocked." Functionally the same situation as 1015: one address crossed a rate threshold and needs to cool down before it’s useful again. Same fix, slow down, and give it time. Confuse a 520 with a 1015, and you’ll spend an afternoon tuning a fingerprint when the real fix was just slowing down. Knowing which one you’re looking at is most of the job.
Once you know whether you’re dealing with detection or just an overloaded server, the next question is what a setup looks like that avoids both problems from the start, and where infrastructure that gets the fingerprint layer right from the beginning actually changes how often you see any of these at all.
Your 2026 scraping checklist
Run your own setup against these five things. Each one is a place teams commonly fall short, and a place the popular options don’t fully solve on their own.
| What to check | How Titan handles this | |
|---|---|---|
| 1 | A real device behind every IP, not just a clean address. If you’re shopping for a proxy provider right now, you’ll hear a lot about scale: more IPs, more countries, higher success rates on paper. BrightData and Oxylabs both compete almost entirely on that basis, and it’s a real advantage, as far as it goes. What it doesn’t solve is the fingerprint problem, because an address on its own doesn’t have one. It’s just a location. If nothing real is behind that address, your script still has to fake the rest of the connection convincingly by itself, which is exactly what detection systems are built to catch. | Titan’s network is built from real user devices whose owners opted in to share bandwidth. So when your request goes out, the fingerprint riding along with the IP is already genuine. Nobody had to fake it afterward. |
| 2 | A connection that behaves like an ordinary one, not a script. Most teams never think to check this, because it happens before their own code even runs. You don’t need to become a protocol expert to fix it, either. You just need a client built to match real browser behavior by default, or a genuine browser session for the pages that actually need one, so your request never announces itself as a script in the first place. | Titan handles the address side of this automatically through its network. Pairing that with a client tuned to behave like a real browser, rather than a default library running its out-of-the-box settings, is what actually closes the loop on both sides of the connection at once. |
| 3 | Browsers spent only where they earn their cost. When a scraper starts failing, the instinct is to reach for a full browser, tools like Playwright or Puppeteer driving real Chrome, on the theory that a real browser can’t get caught. That’s true for the parts of a job that genuinely need one: logging in, running pages that only load content through JavaScript, starting a session. It’s not true for everything. If your team’s answer to failures has been spinning up more browser instances, you’re usually scaling the most expensive part of the system instead of fixing the real problem. | When the IP and device signature are already authentic, fewer pages need a full browser at all, so your browser budget shrinks down to the handful of sessions that genuinely require one. |
| 4 | Traffic routed by what it needs, not by habit. Take a look at how your proxy spend is actually split right now. There’s a good chance almost everything runs through the same, most expensive tier, no matter what each request actually needs. Logging in benefits from a stable residential identity. A list page or detail page often doesn’t. Hitting a public API directly needs no proxy at all. The number worth tracking isn’t cost per gigabyte, it’s cost per successful record: a cheap connection that only works 40% of the time ends up pricier than an expensive one that works 95% of the time, once you count every retry. | This kind of segmenting only works if the network underneath actually supports it, without making you stitch together separate vendors for each tier. Titan’s residential, datacenter, and rotating options all sit on one network instead of needing a different contract for each. |
| 5 | Proof on your own targets before you commit to anything. None of the first four things mean much until you’ve tested them on the sites you actually need, not a generic benchmark. | Titan’s free tier, 2,000 credits a month with no credit card required, exists for exactly this. [Point it at your real targets →] and see what gets through before any of this becomes a line item. |
Run your setup against those five, and the gap between something that occasionally works and something built to work becomes obvious fast.
There’s a separate question worth asking too, and it has nothing to do with fingerprints or IPs: is any of this actually allowed?
What’s actually legal here, and where it’s genuinely still murky
When it comes to the question of “is web scraping actually legal?”, both sides tend to overstate their case. People who scrape for a living tend to say it’s totally settled and harmless. People who run websites tend to say any scraping at all is unauthorized access. Both are oversimplifying. Here’s the version that’s actually useful: what’s settled, what’s still murky, and what it means for picking infrastructure.
What’s actually settled: collecting data that’s genuinely public, meaning you never logged in, never made an account, and never got past any wall to see it, has held up again and again as fundamentally different from breaking into a system. The clearest example is hiQ Labs v. LinkedIn, where the Ninth Circuit ruled, and reaffirmed on appeal, that scraping data LinkedIn had made publicly accessible did not violate the Computer Fraud and Abuse Act, since no authorization was ever required to view it in the first place. The Supreme Court’s decision in Van Buren v. United States reinforced the same principle in a different context, narrowing the CFAA to cases where someone actually bypasses a real access restriction rather than just violating a policy about how public access should be used. Public means public. Automating the collection doesn’t turn it into unauthorized access.
What’s still murky: a site’s terms of service can create real exposure that has nothing to do with whether your access counted as “authorized.” hiQ is the clearest cautionary tale here too. The company won decisively on the CFAA question and still lost the war, because a separate 2022 summary judgment found hiQ liable for breach of contract for violating LinkedIn’s User Agreement, an entirely different legal theory that the CFAA ruling never touched. You can be completely fine on the authorization question and still be bound by a contract, just because you created an account at some point, even for something unrelated. Whether you were logged in or logged out when you scraped matters a lot here, even when the data you collected is exactly the same either way. A related 2024 ruling in Meta’s case against Bright Data followed a similar pattern, turning on whether the scraping happened while logged into an account bound by terms, not on whether the underlying data was public.
Outside the US, things are getting stricter too. Newer data protection and AI rules increasingly expect you to show where your data actually came from, not just confirm it was public. Under the EU AI Act, providers of general-purpose AI models are now required to publish a detailed summary of the data used to train their models, including the sources and how it was collected, following a standardized template from the EU’s AI Office. Saying “we scraped it” isn’t enough on its own anymore, especially if what you’re building touches European markets.
Given that, the real question to ask about any infrastructure you’re using isn’t “is this legal.” It’s “does this protect me on the parts I actually control.” Three things worth checking:
- Does the provider ever create logged-in sessions or accounts on your behalf, the kind that could bind you to terms you never meant to accept?
- Do they know, and can they tell you, where their own IPs and nodes actually come from?
- If a client or compliance team ever asks where your data came from, is there a real answer?
On that last point, this is where Titan’s sourcing actually matters. Every node on the network comes from a device whose owner opted in and gets paid for sharing bandwidth, so that history exists from day one instead of getting pieced together later if someone asks. Nothing on the network comes from a hijacked connection or a device taken without consent. What none of that changes is your own side of things: your targets, whether you’re logged in or out, what you do with the data once you have it. That part is still yours to get right, and if you’re running this at real production scale, an actual lawyer is worth more than a guess.
You don’t have to take any of this on faith. The free tier, 2,000 credits a month, no credit card required, is enough to test all five checklist items against your actual targets before you commit to anything at scale.
Stop treating 520s and 1015s like random errors.
If blocked IPs, failed requests, and unstable pipelines are slowing your data team down, Titan can help rebuild the collection layer with residential IPs and real-device authenticity.
