Build a scraper that handles 402 responses and AI crawler rules

For years a scraper met three kinds of response: success, an error, or a block that looked like an error. That is changing. Site owners now have tools that answer automated traffic with an explicit, machine-readable message: this is not free, here is the price, or here is how to contact us.

If you run scrapers for a business, that is good news for design, not just a compliance chore. A block that says what it is can be handled in code. A block that looks like a timeout cannot.

This post covers what changed, based on Cloudflare's published changelog, and a response-handling design we would use for a production scraper.

What changed on the site side

Cloudflare's AI Crawl Control changelog (read on 2026-11-03) records the relevant steps:

A few details matter for scraper authors. Errors carry a crawler-error header with a code, /robots.txt, /sitemap.xml and a few other well-known paths are always free, and the product was still labelled beta in the documentation we read. This is one vendor, so treat it as a pattern rather than the whole web. We could not open the individual Pay Per Crawl how-to pages in this run, so the payment flow itself is not described here.

Who this affects, and who it does not

Much of this targets AI training and AI search crawlers, which are verified bots signing their requests. A company scraping prices for its own analytics is a different case, but the same signals can reach you: a 402, a content-signals line in robots.txt, or a terms page that forbids automated access.

Do not assume a rule only applies to someone else. Assume your scraper will meet these responses and decide ahead of time what it does.

A response classifier instead of a retry loop

The common mistake is treating every non-200 as "retry with backoff". That is wrong for a 402 and wasteful for a 403. Put a classifier between the HTTP client and the pipeline:

from enum import Enum

class Verdict(Enum):
    OK = "ok"
    RETRY = "retry"          # transient: 429, 5xx, timeouts
    PAID = "paid"            # 402: needs a commercial decision
    FORBIDDEN = "forbidden"  # 401/403: stop, do not evade
    GONE = "gone"            # 404/410

def classify(resp) -> Verdict:
    s = resp.status_code
    if s == 200:
        return Verdict.OK
    if s == 402:
        return Verdict.PAID
    if s in (401, 403):
        return Verdict.FORBIDDEN
    if s in (404, 410):
        return Verdict.GONE
    if s == 429 or s >= 500:
        return Verdict.RETRY
    return Verdict.FORBIDDEN

Then give each verdict a policy:

• OK: parse and validate as usual.
• RETRY: exponential backoff with a cap, and honour Retry-After if present.
• PAID: do not retry. Log the host, the URL pattern and any price or crawler-error header, then route it to a person. Whether to pay, license the data or drop the source is a business decision.
• FORBIDDEN: mark the URL pattern as off limits for a long cool-down. Do not rotate proxies or user agents to get around it.
• GONE: record it and stop requesting.

Check robots.txt and signals before the first request

Fetch /robots.txt once per host per day, cache it, and evaluate it for your own user agent before queueing URLs. Python's standard library has urllib.robotparser for the basic rules. It does not understand newer directives such as content signals, so read those lines yourself and store them next to the host record.

Identify yourself. A descriptive User-Agent with a contact address costs nothing and gives a site owner a way to reach you instead of blocking you. Keep concurrency per host low and spread requests out.

Make blocked sources visible

Treat a blocked source like any other data quality event. The failures that hurt in scraping pipelines are the silent ones, where a source quietly stops producing rows. Track, per source and per day:

1. Share of requests by verdict.
2. Rows produced versus the trailing average.
3. First date a 402 or 403 appeared.

Alert when the share of PAID or FORBIDDEN responses for a source crosses a threshold you choose, for example anything above zero for a source you rely on. Then the business hears about a lost source in a day, not at the next report.

When to buy, license or switch

When a source answers with a price or a refusal, the options are not only "pay" or "give up":

• Official API or licensed feed. Often cheaper than engineering time spent on a fragile scrape.
• A different source for the same field, such as a government dataset, a sitemap or structured data the site publishes for you.
• Drop the field if it is not worth the cost.

Put a number on it. Multiply the pages you need per month by the quoted price and compare it with the engineering and maintenance time for the scrape. We would run that comparison when scoping data extraction work; it tends to favour a licensed feed when there are only a few high-value sources.

Takeaway

Add a response classifier, a robots.txt and signals check, and a per-source verdict dashboard before you add the next target site. When a site says no or names a price, your scraper should record it, stop, and tell a person. Last checked 2026-11-03; vendor behaviour in this area is changing, so verify against the current documentation.