Stop Scraping Bots With Cloudflare and Vercel

Modern scraping bots do not announce themselves. They spoof a real iPhone or Chrome browser and arrive from a datacenter, which is precisely the tell. You stop them not by reading the user agent, which any client can set to anything, but by combining signals a bot cannot cheaply fake: the network an address truly belongs to (its ASN), the fingerprint its TLS handshake leaves behind, and how it behaves over time. Cloudflare and Vercel, the platforms you very likely already run, ship those exact tools: bot scoring, ASN and fingerprint rules, rate limits, and invisible challenges on your sensitive endpoints. Layer Cloudflare's edge in front of Vercel's firewall and you kill the cheap, high-volume majority outright while catching the clever remainder at the door, without locking out real customers or the search crawlers you want.

The short version

The bot is trying to look like your best customer. The scrapers that cost money present an ordinary phone or desktop browser, spread across many addresses, and pace themselves like a human, so "block the obvious ones" gets you nowhere.
The user agent is a lie you cannot fix by reading it. It is a string the client picks for itself; block one value and the attacker types another for free.
The durable signals are network, fingerprint, and behavior. Where a request truly comes from (ASN), what its TLS handshake reveals, and how it acts over time are expensive to fake, which is exactly why they work.
The iPhone-from-a-datacenter mismatch is your cleanest rule. A request claiming a consumer phone but originating in a cloud provider's network is almost never a real phone.
Cloudflare and Vercel already carry the tools. Bot scoring, ASN and fingerprint rules, rate limits, managed challenges, and invisible checks on logins and checkout ship on the platforms you run.
Layered beats single. Cloudflare at the edge turns away cheap volume before it costs a serverless invocation; Vercel's firewall and BotID guard the specific endpoints that matter most.

What does a "smart" scraping bot actually look like?

The crude bots make themselves easy: an obvious automated user agent, one URL hammered, simple to swat. The ones that cost you money do the opposite. They present a perfectly ordinary phone or desktop browser as their identity, spread their requests across large pools of addresses so no single one looks busy, and pace themselves to resemble a human clicking around, with the more sophisticated ones driving a real headless browser so the page even runs JavaScript. And they go straight for what is valuable: your live prices, your product data, your availability, your written content.

The point is that the bot is not trying to look like a bot. It is trying to look exactly like your best customer, which is why the instinct to "just block the bad ones by name" never gets you very far. That lifted data is not an abstract loss: a competitor with your live prices has your strategy, and a catalog copied wholesale can be republished or fed into a model you never agreed to train, which is part of why exposed data touches the obligations in privacy and CCPA for a website.

How do you spot a phone that lives in a datacenter?

Here is the specific pattern that gives them away. A real iPhone reaches your site over a mobile carrier or a home broadband connection. A request that presents itself as an iPhone but originates inside a major cloud provider's datacenter is almost never a real iPhone, because real people do not browse your store from inside a server rack.

Every address on the internet belongs to a network block identified by an ASN, the number that says which provider an IP comes from, whether a home ISP, a mobile carrier, or a datacenter, and that single fact is one of the cleanest signals you have. When the user agent claims a consumer device but the ASN belongs to a large cloud network, including the big Chinese clouds such as Alibaba Cloud, Tencent, and Huawei among many datacenter networks worldwide, the outside and the inside of the request disagree. A consumer device on the surface, a server farm underneath. That mismatch is the thread you pull, and noticing it usually starts with actually watching your traffic, the discipline behind our observability for small apps guide.

Why does blocking by user agent never work?

It is tempting to read that user agent and simply ban it. But a user agent is just a line of text the client chooses for itself, and anyone can set it to anything. Block a scraping library's default and it becomes "iPhone." Block that and it becomes "Chrome on Windows." You are playing whack-a-mole against a string the attacker rewrites for free, while risking real customers whose browsers happen to share that text.

Durable protection comes from combining signals the bot cannot cheaply fake: where the request truly comes from, what its connection actually looks like underneath the claims, and how it behaves over time. That is exactly what the platforms below are built to do.

What does Cloudflare give you?

Cloudflare sits in front of your site and sees a vast slice of the internet's traffic, which is what makes its detection strong. Its bot detection features, roughly from simplest to most advanced:

Bot scoring. Bot Fight Mode, Super Bot Fight Mode, and Bot Management on higher tiers score each request on how automated it looks, using machine learning trained across Cloudflare's whole network. You can challenge or block automated-scoring traffic while letting verified good bots like Googlebot pass untouched.
WAF custom rules that combine signals. This is where you encode the iPhone-from-a-datacenter rule in plain logic: if the user agent looks like a mobile device and the request comes from a datacenter ASN you never sell to, challenge or block it. One rule can weigh ASN, country, bot score, threat score, user agent, and path together.
ASN and country targeting. Cloudflare lets you match on the exact network an address belongs to and on geography, so you can challenge entire datacenter networks that only ever send you scrapers, or regions you do not serve, without disturbing real visitors.
TLS fingerprinting (JA3 and JA4). Every client leaves a fingerprint in the way it negotiates its secure connection, and that fingerprint reveals the real software making the request no matter what user agent it claims. A scraping tool dressed up as Safari still has the fingerprint of the tool, and Cloudflare can act on the truth rather than the costume.
Managed Challenge and Turnstile. Instead of a hard block, present a lightweight challenge that real browsers pass invisibly and bots fail. Turnstile is the privacy-friendly alternative to the old image puzzles, placed on logins, signups, and checkout to stop automated abuse without annoying a human.
Rate limiting. Cap how many requests a single source can make to a sensitive path, so a scraper machine-gunning your product pages or an API endpoint gets throttled while a real shopper never comes close to the limit.
IP reputation. Cloudflare carries a reputation signal for addresses with a history of abuse across its network, so known-bad sources can be challenged automatically before they do anything.
Blocking AI scrapers and crawlers. Cloudflare added straightforward controls to block the recent wave of AI training crawlers, plus tools that feed misbehaving crawlers decoy content instead of your real data. If you would rather your catalog and writing not quietly become training material, this is a real lever.
Managed rulesets and DDoS protection. Underneath all of it is baseline protection that absorbs volumetric attacks and known exploit patterns before they ever reach your origin.

What does Vercel give you?

If your app runs on Vercel, you have a capable firewall at the edge, and the Vercel Firewall has grown into a serious bot-protection layer:

Custom rules on the signals that matter. You write rules on path, IP, geography, user agent, request headers, and TLS fingerprint to challenge, block, or rate-limit traffic, the same combine-the-signals approach, native to the platform your app already lives on.
Attack Challenge Mode. A single switch that puts a verification step in front of suspicious traffic during a bot surge, separating real browsers from automation without taking the site offline for anyone.
BotID, invisible bot detection. An invisible challenge you can put in front of your most sensitive actions, login, signup, checkout, and critical API routes, that catches sophisticated bots without ever showing a human a puzzle.
Rate limiting and IP deny lists. Throttle or block abusive sources per route, so one endpoint under attack does not drag down the rest.
Automatic DDoS mitigation. Baseline volumetric protection that comes with the platform rather than as an upsell.

Which endpoints deserve the strongest guard is the same question we work through for any privileged surface in the secure admin panel guide: protect the login, the checkout, and the API before you worry about the marketing pages.

Why put Cloudflare in front of Vercel?

For many projects the strongest setup is layered. Cloudflare sits at the very edge handling DNS, the firewall, the ASN and bot-score rules, and rate limiting, so the bulk of bad traffic is turned away before it costs a single serverless invocation on Vercel. Vercel's own firewall and BotID then guard the endpoints that matter most, so cheap scraping dies at the Cloudflare edge while anything clever enough to slip through meets BotID at the door of your checkout or API.

Fronting Vercel with Cloudflare has setup details that must be right to avoid breaking caching or origin verification, which is why it rewards being done deliberately rather than by flipping switches and hoping. We run production backends on Vercel daily, so this is familiar ground, and the edge and webhook discipline behind it is the same as in our guide to the Revolut Merchant API on Vercel.

How do you roll this out safely, step by step?

Bot rules can block real traffic, so the order you turn things on matters as much as the rules themselves.

Read your real logs first. Look at which ASNs, paths, and fingerprints are hitting you and at what rhythm, so the rules target real offenders instead of a guess.
Start every rule in log-only mode. Cloudflare and Vercel both let a rule observe and count without blocking. Run it against live traffic and confirm it catches the bots and spares the humans before it does anything.
Turn on the highest-value rule first. Usually that is the datacenter-pretending-to-be-a-phone pattern, promoted from log-only to challenge once the counts look clean.
Add rate limits on the paths under pressure. Set the cap well above what a real shopper hits and watch the throttle bite the scrapers, not the customers.
Put an invisible challenge on the sensitive actions. Turnstile or BotID on login, signup, and checkout, so automated abuse fails there without a human ever seeing a puzzle.
Layer the edge only once the origin rules hold. Front Vercel with Cloudflare after the app-level rules are proven, checking caching and origin verification still behave.
Measure before and after, then keep tuning. Compare the graphs, confirm no real customer or good crawler got caught, and revisit as the bots adapt, because they will.

A pre-launch checklist

Have you looked at real logs to see which ASNs, paths, and fingerprints are actually hitting you?
Is every new rule running in log-only mode before it blocks anything?
Does the datacenter-plus-mobile-user-agent rule spare the legitimate cloud traffic you do want (payment webhooks, monitoring, your own jobs)?
Are rate limits set above real customer behavior, not at it?
Is there an invisible challenge on login, signup, checkout, and critical API routes?
Are verified good bots like Googlebot explicitly allowed to pass?
If you front Vercel with Cloudflare, have you confirmed caching and origin verification still work?

Common pitfalls

Blocking a whole country or cloud in one swing. A blunt rule banning an entire region or every datacenter address feels decisive and quietly takes out real traffic: customers on a VPN, your own uptime monitor, a payment provider's webhook, a partner's integration. Precision beats force, which is why the mismatch rule (consumer user agent plus datacenter ASN) is safer than "block the cloud."

Shipping rules straight to blocking. A rule that looks obviously correct on paper can still catch a slice of real visitors you did not anticipate. Running it in log-only mode against live traffic first is the difference between a tuned filter and a self-inflicted outage.

Protecting the marketing pages and forgetting the API. Scrapers go where the value is, and the value is usually a JSON endpoint or a search route, not the homepage. If your rules guard what is easy to see and leave the data endpoints open, you have decorated the front door and left the loading dock unlocked.

A concrete case. A store we looked at had a creeping serverless bill and analytics that no longer matched reality, with traffic spiking at odd hours that never turned into sales. The logs told the story: tens of thousands of requests an hour, each presenting as an iPhone, all from a handful of overseas datacenter networks, walking the catalog and the price API in a rhythm no human keeps. The owner's instinct had been to block the country, which would have cut off traveling customers and their own monitoring. Instead we wrote the mismatch rule, mobile user agent plus datacenter ASN, ran it in log-only mode for a few days to confirm it caught the scraper and nothing else, then promoted it to a challenge and rate-limited the price endpoint. The junk traffic collapsed, the bill settled, the analytics matched real behavior again, and not one genuine shopper or crawler was turned away. The lesson was not "block harder." It was that the outside and the inside of those requests disagreed, and the rule only had to notice.

What can protection realistically do, and what can't it?

No setup blocks every bot, and anyone who promises otherwise is selling something. The most determined scrapers rent residential addresses and drive real browsers, which is expensive for them, and that expense is the point. Good protection kills the cheap, high-volume majority outright and forces the rest to spend, while challenges and fingerprinting catch most of what remains. The goal is a poor target, not a flawless wall, reached without ever getting in the way of a real customer or a search engine you want to be found by.

Keeping infrastructure fast, affordable, and defended from abuse is everyday work for us, not a side service. We run real production software on Vercel, including the customs-invoice.com compliance platform and storefronts with live payments such as LeO-Optic, and the same care for clean, owned, well-defended infrastructure runs through our eCommerce work as WooSmiths. If you are weighing how much of your stack to keep under your own control, our piece on owning your project on Vercel makes the broader case. And if your traffic graphs have a shape you cannot explain, or you suspect someone is scraping your prices while your bill pays for the privilege, tell me what you are seeing and I will map out the Cloudflare and Vercel protection that fits, without locking out the customers and crawlers you want.

FAQ

How do I know if my site is being scraped?

The signs show up in the shape of your traffic first. Requests climb at odd hours with no matching sales, your hosting bill creeps up, analytics stop lining up with real behavior, and one request pattern repeats thousands of times an hour against your product or price pages. Pull your access logs and look for a single user agent arriving from many datacenter addresses, or a machine-even rhythm no human keeps. Heavy volume with zero conversions is the tell that something automated is walking your catalog.

Why can't I just block the bad bots by user agent?

Because the user agent is a label the client writes for itself, not a fact about the client. A scraper can set it to any value, so blocking one string simply prompts the attacker to type a different one, at no cost, while you risk banning real visitors whose browser sends the same text. Effective blocking ignores the label and looks at things the bot cannot cheaply change: the network its address belongs to, the fingerprint its secure connection leaves, and its behavior over time. Those are the signals Cloudflare and Vercel let you act on.

What is an ASN, and why does it matter for bot blocking?

An ASN, or Autonomous System Number, identifies the network operator an IP address belongs to, telling you whether a request comes from a home ISP, a mobile carrier, or a datacenter. It matters because it exposes the disagreement at the heart of a smart scraper: a request can claim to be an iPhone, but if its ASN belongs to a cloud server farm, the claim is false, because real customers do not browse from inside a datacenter. Matching on ASN lets you challenge entire networks that only ever send automation while leaving genuine visitors untouched.

Can Cloudflare and Vercel stop every scraper?

No, and any tool that claims to is overselling. The most determined operators rent residential addresses and drive real browsers, which defeats the cheap signals, but doing so is expensive and slow, and that cost is the objective. Solid protection eliminates the high-volume majority outright and forces the rest to spend real money, while invisible challenges and fingerprinting catch most of what is left. You are not building a perfect wall; you are making your site an unattractive target and defending the endpoints and data that matter.

Will bot protection block Google or my real customers?

It should not, and avoiding that is the whole discipline of setting it up well. Verified good bots such as Googlebot can be explicitly allowed through, and the sharpest rules target mismatches, like a consumer user agent from a datacenter, rather than blunt bans on whole countries or clouds that sweep up real people. Run every rule in observe-only mode against live traffic first, confirm it catches automation and spares humans, and keep a quick path to loosen anything too tight. Done that way, customers and crawlers never notice.

Do I need both Cloudflare and Vercel, or is one enough?

Either can protect you alone, but layering them is stronger for most projects. Cloudflare at the edge turns away cheap, high-volume scraping before it costs a serverless invocation, handling DNS, the firewall, ASN and bot-score rules, and rate limiting. Vercel's firewall and BotID then guard the endpoints that matter most, catching anything clever enough to slip through. The one caution is that fronting Vercel with Cloudflare has caching and origin-verification details that need to be right, so it rewards being done deliberately.

Smart Scraping Bots That Fake an iPhone: How Cloudflare and Vercel Fight Back