How to build a web scraper that survives site changes

Most scrapers do not fail loudly. A selector stops matching, a field comes back empty, and the pipeline keeps running. Weeks later someone notices the dashboard is built on half-empty rows.

The fix is rarely a smarter parser. It is a design that assumes the target site will change, and makes that change visible within hours instead of weeks. This post lays out the structure we would use for a scraper a business depends on.

Start by asking whether you need to scrape at all

Before writing any parsing code, check the cheaper sources of the same data, in this order:

1. An official API or data export. Even a limited one is more stable than HTML.
2. A structured feed. Sitemaps, RSS, JSON-LD blocks and <script type="application/json"> payloads often carry the data you want in a cleaner form than the visible markup.
3. The site's own internal JSON endpoints. Many pages load data from a JSON request. These change less often than layout, but they are not a published contract, so treat them as fragile and check the terms of use.
4. Rendered HTML. The last resort, and the most likely to break.

Each step down this list costs you stability. Write down why you are on step 3 or 4, because the next engineer will ask.

Check the rules before the first request

Scraping is a legal and contractual question as much as a technical one, and we are not lawyers. What we can say as engineers is that the checks below cost almost nothing and belong in the code from day one.

• Read the site's terms of use and robots.txt. Respect disallow rules and any crawl-delay.
• Do not collect personal data you have no basis to hold.
• Identify your client with a real contact address in the user agent.
• Rate-limit per host, and back off on 429 and 5xx responses.

Python ships a robots parser, so there is no reason to skip it:

from urllib.robotparser import RobotFileParser

def allowed(url: str, robots_url: str, agent: str = "ForkLabsBot") -> bool:
    rp = RobotFileParser()
    rp.set_url(robots_url)
    rp.read()
    return rp.can_fetch(agent, url)

Cache the parsed file per host instead of re-reading it on every call. If a client needs data from a site that disallows automated access, the answer is to ask the site owner for access or a licence, not to engineer around the block.

Build a fetch ladder, not a single fetcher

Different pages need different amounts of machinery. A browser for every URL is slow and expensive; plain HTTP for every URL fails on pages that render client-side. A ladder lets each URL use the cheapest tier that works.

The scraper tries tier 1, checks the result against the schema (next section), and escalates only on failure. Log which tier each domain settles on. That log becomes a cost report: if a domain moves from tier 1 to tier 2, your bill just changed and you want to know why.

Validate every record against a schema

This is the part that prevents silent failure. Define what a good record looks like and reject anything else before it reaches storage.

from pydantic import BaseModel, HttpUrl, field_validator

class Product(BaseModel):
    url: HttpUrl
    name: str
    price: float
    currency: str

    @field_validator("name")
    @classmethod
    def not_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("empty name")
        return v

    @field_validator("price")
    @classmethod
    def sane_price(cls, v: float) -> float:
        if not 0 < v < 1_000_000:
            raise ValueError("price out of range")
        return v

Validation catches broken records. It does not catch a selector that still matches, but now points at the wrong element, such as a "was" price instead of the current price. For that, add cheap sanity checks across the batch:

• Compare the share of records with each field populated against the previous run.
• Compare the median of numeric fields against the previous run, and flag large jumps.
• Count records per source page. A listing page that used to yield 24 items and now yields 0 or 1 is a layout change.

Alert on drift, quarantine bad output

When a batch fails these checks, do not overwrite the last good dataset. Write the new batch to a quarantine location and keep serving the previous one. Then alert a person with the specifics: which domain, which field, the old and new fill rate, and a saved copy of an example page.

Saving raw HTML for failed pages matters more than it sounds. It lets you repair the parser against the exact page that broke, and it lets you write a regression test from it. Keep fixtures for each target site, and run the parser against them in CI so a refactor does not reintroduce an old bug.

A minimal monitoring set that has worked well in practice:

• A per-domain success rate, plotted over time.
• A per-field fill rate, with an alert threshold you choose after watching a few weeks of normal data.
• A daily canary: one known-good URL per domain, fetched and parsed end to end.

Make selectors boring

When you must parse HTML, prefer anchors that the site has reason to keep stable: itemprop and JSON-LD attributes, data-* test hooks, semantic tags and visible labels. Avoid generated class names (css-1x2y3z) and deep positional paths (div > div:nth-child(3) > span).

Keep two or three fallback selectors per field, ordered from most to least stable, and log which one matched. A shift from the first to the second selector is an early warning that costs nothing to collect.

Takeaway

A scraper you can rely on is mostly not scraping code. It is a fetch ladder that controls cost, a schema that rejects bad records, batch-level checks that notice drift, and a quarantine step that protects the last good data. If you are scoping a data extraction project, ask your vendor or your team how they will know when the target site changes. If the answer is "we will notice", the pipeline will fail quietly sooner or later.

At Fork Labs we build extraction pipelines this way, with the monitoring designed in rather than added after the first outage.