How to Stop Product Scraping with Cloudflare

How do you tell scraping apart from normal traffic?

Before you write a single rule, you need to see the problem clearly. Scrapers behave differently from shoppers and from legitimate search engines, and Cloudflare gives you the telemetry to spot the difference. Start in the Security Analytics and Bot Analytics views and look at how traffic is distributed across bot scores, source networks and request paths. Genuine human sessions tend to be score-rich, browse a handful of pages and pull images, CSS and JavaScript. Scrapers often request only the HTML of product and category pages, in tight sequences, from a narrow set of networks.

A few signals are especially telling. Sequential or enumerated URL access — hitting product IDs or paginated category pages in order — is a strong scraping tell, as is a high request rate concentrated on catalogue endpoints rather than spread across the site. Traffic originating from datacentre ASNs (cloud providers, hosting companies and proxy networks) rather than residential ISPs is suspicious for a consumer store, and headless-browser fingerprints or missing, generic or rotating User-Agent strings point the same way.

Resist the urge to block on day one. Spend a short period in observation mode, tagging the patterns you see, so your eventual rules target real abuse and not a misread of normal behaviour. A Cloudflare Audit is a good way to baseline what is actually hitting your catalogue before you start changing anything, so you can measure whether your later rules are working.

• High request rates focused on product and category pages
• Sequential or enumerated URL crawling (IDs, pagination)
• Datacentre ASNs and proxy networks rather than residential ISPs
• Headless-browser fingerprints and missing or rotating User-Agents
• Sessions that skip images, CSS and JavaScript and pull HTML only

Why layered friction beats a single block rule

There is no single switch that stops scraping, and any vendor who promises one is overselling. A determined operator can rotate IP addresses, swap User-Agents, render JavaScript and even solve some challenges. The realistic goal is to make scraping slow, costly and unreliable enough that the economics stop working — most operators are scraping many sites and will deprioritise the one that fights back.

Layering matters because each control catches a different slice of traffic. Bot management scores requests and lets you act on the obviously automated ones. Rate limiting caps how fast any single source can pull catalogue pages, blunting bulk harvesting even when the source looks plausible. Managed challenges and Turnstile add interactive friction for the in-between cases that are suspicious but not clearly malicious. Used together, these turn a cheap, fast scrape into an expensive, throttled crawl.

Order your rollout from least disruptive to most. Begin with logging-only or low-impact actions, confirm you are not catching shoppers or search engines, then escalate the action — from log, to managed challenge, to block — as your confidence grows. This staged approach is the heart of effective Cloudflare Bot Protection, and it is far safer than flipping everything to block at once.

• Bot management to score and act on automated traffic
• Rate limiting to cap how fast catalogue pages can be pulled
• Managed Challenge or Turnstile for ambiguous, mid-score traffic
• Staged actions: log first, then challenge, then block

How should you configure rate limiting for catalogue pages?

Rate limiting is the most reliable single lever against bulk scraping because it targets the behaviour — pulling many pages quickly — rather than trying to identify the actor. In the Cloudflare dashboard, create rate-limiting rules that match your catalogue paths (for example, requests to your product and category URL patterns) and set a threshold that comfortably exceeds normal human browsing but trips on automated harvesting.

Choose your counting key carefully. Rate limiting by IP address is the default, but sophisticated scrapers rotate IPs, so where your plan allows, key on a combination such as IP plus User-Agent, or on a JA4/JA3 fingerprint, to make rotation less effective. Set the action to a managed challenge rather than an outright block at first; this lets a real shopper who briefly trips the limit continue, while a headless scraper stalls. Keep windows short and thresholds tuned to your real traffic — too tight and you frustrate genuine browsers, too loose and you let scrapers through.

Remember that rate limiting protects more than HTML pages. Your search endpoint, autocomplete API, GraphQL or REST product APIs and any pricing or stock feeds are common scraping targets and deserve their own, often stricter, limits. Pair these rules with sensible caching so that legitimate bursts of traffic are served from the edge and never reach your origin, which also helps protect origin capacity during an aggressive crawl.

How do you protect SEO while blocking scrapers?

The biggest risk in any anti-scraping project is collateral damage to search visibility. Blocking Googlebot, Bingbot or other legitimate crawlers to stop scrapers is a self-inflicted wound that can quietly erode rankings and revenue. Your rules must distinguish verified search engines from impostors that merely claim to be them in their User-Agent.

Cloudflare helps here with verified bot detection: it validates well-known crawlers by checking that the request genuinely originates from the search engine's published networks, not just that it carries the right User-Agent string. Build your rules so that verified bots are explicitly allowed through before any challenge or block logic runs. A common, robust pattern is to allow verified search-engine and other trusted bots, allow scored-human traffic, and only then apply challenges or blocks to the remaining automated and unverified requests.

Test before and after every change. Use Google Search Console's URL inspection and crawl stats to confirm Googlebot is still fetching pages, and watch your Cloudflare logs for any verified crawler that is being challenged. If you run product feeds for Google Shopping or partner integrations, allowlist those specific systems by IP range or shared secret so your anti-scraping rules never starve a feed your business depends on.

Where do AI crawlers fit into your scraping policy?

A growing share of catalogue scraping now comes from AI companies harvesting product data to train or feed models. Some identify themselves honestly through documented crawlers; others are far less transparent. You need an explicit policy on what you will allow, because the trade-offs differ from classic competitor scraping — some AI crawlers may drive referral traffic, while others simply take your catalogue for nothing.

Cloudflare exposes controls aimed specifically at this traffic. You can use its managed rules and bot categories to identify and challenge or block AI crawlers, and combine that with a clear robots.txt declaration of what is and is not permitted. Robots.txt is a request, not an enforcement mechanism, so honest crawlers will respect it while dishonest ones will not — which is exactly why the edge-level controls matter as the enforcement layer behind the policy.

Decide deliberately rather than by default. If you want product pages indexed by AI search experiences but not bulk-harvested for training, your AI Crawler Control configuration should reflect that nuance, allowing the crawlers you value and adding friction to the rest. Review the list periodically, because new crawlers appear regularly and the right answer for your brand may change as those services evolve.

What does a sensible rollout and monitoring routine look like?

Treat anti-scraping as an ongoing programme, not a one-off configuration. Start by deploying your rules in a low-impact mode — logging or managed challenge — and watch the Security Events and Bot Analytics dashboards closely for a week or two. You are looking for two things: that you are catching the scraping patterns you identified, and that you are not catching shoppers, checkout flows, payment callbacks or legitimate bots.

Once you are confident, escalate the action on your highest-confidence rules from challenge to block, and tighten thresholds where the data supports it. Keep a small allowlist of systems you must never interfere with: your own monitoring, uptime checks, payment provider webhooks, marketplace integrations and verified search engines. Document every rule and why it exists, so a future change does not silently undo your protection.

Finally, accept that scrapers adapt, so your defences must too. Schedule periodic reviews, watch for new networks or fingerprints appearing in your logs, and adjust. Cloudflare is one important layer, but it works best alongside good development hygiene — sensible pagination, not exposing internal IDs needlessly, and protecting APIs at the application level too. If keeping on top of this is more than your team can carry, ongoing tuning and review is exactly the kind of thing a managed service or a periodic audit is designed to handle.