Web Scraping Retry Patterns: Backoff, Jitter, and When to Stop

Most scraping retry logic is either too timid or too stubborn.

Too timid means one timeout kills the job.

Too stubborn means the scraper keeps retrying a blocked target, multiplies traffic, and turns a recoverable hiccup into a guaranteed ban.

The right approach is simple:

• retry transient failures
• back off between attempts
• add jitter so every worker does not retry together
• stop when the target is clearly telling you to stop

Why retries matter in scraping

A scraper does not only fail because your parser is wrong. It also fails because the network and the target are imperfect:

• connection resets
• DNS hiccups
• slow responses
• 429 Too Many Requests
• 502, 503, and 504
• temporary bot defenses

Search results for modern retry guidance consistently surface the same advice from systems literature and scraper debugging guides: use exponential backoff, add jitter, and do not retry everything blindly. That is the core pattern we will implement here.

The retry decision table

Not every failure deserves another attempt.

That last row is the one many scrapers get wrong. If the site has switched you to a challenge flow, ten rapid retries only make the evidence stronger.

Pattern 1: Exponential backoff

Exponential backoff means each retry waits longer than the previous one.

Typical delays look like:

• attempt 1 -> wait 1 second
• attempt 2 -> wait 2 seconds
• attempt 3 -> wait 4 seconds
• attempt 4 -> wait 8 seconds

That protects both you and the target.

def backoff_delay(attempt: int, base: float = 1.0, cap: float = 30.0) -> float:
    return min(cap, base * (2 ** max(0, attempt - 1)))

This is the baseline. It is already much better than time.sleep(1) after every failure.

Pattern 2: Add jitter

Jitter means introducing randomness into the wait time.

Why it matters:

• multiple workers may fail at the same time
• if they all retry on the same schedule, they create a retry storm
• jitter spreads that traffic out

import random


def backoff_with_jitter(attempt: int, base: float = 1.0, cap: float = 30.0) -> float:
    upper = min(cap, base * (2 ** max(0, attempt - 1)))
    return random.uniform(0, upper)

This "full jitter" approach is easy to reason about and works well for scrapers.

Pattern 3: Separate retryable and terminal failures

This is where scraping code becomes production-friendly.

from __future__ import annotations

import random
import time

import requests

TIMEOUT = (10, 30)
RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}

session = requests.Session()
session.headers.update(
    {
        "User-Agent": (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/126.0.0.0 Safari/537.36"
        )
    }
)


def sleep_for_retry(attempt: int, *, base: float = 1.5, cap: float = 45.0) -> None:
    upper = min(cap, base * (2 ** (attempt - 1)))
    time.sleep(random.uniform(0.0, upper))


def fetch_with_retries(url: str, attempts: int = 5) -> requests.Response:
    last_error = None

    for attempt in range(1, attempts + 1):
        try:
            response = session.get(url, timeout=TIMEOUT)

            if response.status_code in RETRYABLE_STATUS_CODES:
                last_error = RuntimeError(f"retryable status: {response.status_code}")
                if attempt == attempts:
                    break
                sleep_for_retry(attempt)
                continue

            if response.status_code in {403, 404}:
                response.raise_for_status()

            response.raise_for_status()
            return response

        except (requests.ConnectTimeout, requests.ReadTimeout, requests.ConnectionError) as exc:
            last_error = exc
            if attempt == attempts:
                break
            sleep_for_retry(attempt)
        except requests.HTTPError as exc:
            raise exc

    raise RuntimeError(f"Failed after {attempts} attempts: {last_error}")

Notice the key distinction:

• transport flakiness gets retries
• hard application errors do not

Pattern 4: Respect Retry-After when you get it

Some rate-limited responses include a Retry-After header. If the target gives you a concrete wait time, use it.

def parse_retry_after(response: requests.Response) -> float | None:
    retry_after = response.headers.get("Retry-After")
    if not retry_after:
        return None
    try:
        return float(retry_after)
    except ValueError:
        return None

Then inside the retry block:

if response.status_code == 429:
    delay = parse_retry_after(response)
    if delay is None:
        delay = random.uniform(5.0, 15.0)
    time.sleep(delay)
    continue

This is one of the simplest ways to behave like a less annoying client.

When to stop retrying

This is the part that deserves more attention than it gets.

Stop retrying when:

• the status is 403 and stays 403
• the response is a bot challenge page instead of the real HTML
• the page structure changed and selectors no longer match
• the target is down long enough that the job’s freshness window has expired

Here is the practical rule:

Retries are for uncertainty.

If the target is giving you a consistent answer, believe it.

A practical scraper wrapper

def get_html(url: str) -> str:
    response = fetch_with_retries(url, attempts=5)
    html = response.text

    lowered = html.lower()
    if "captcha" in lowered or "unusual traffic" in lowered:
        raise RuntimeError("Bot challenge detected; stop and slow the crawl")

    if len(html) < 500:
        raise RuntimeError("Response body is suspiciously small")

    return html

This is where retry policy and content validation meet. A technically successful response can still be a scraping failure.

Comparison of common retry strategies

If you only adopt one pattern from this post, make it exponential backoff plus jitter.

Where ProxiesAPI fits

Retry logic and proxies solve different problems.

Retries solve:

• transient failures
• temporary overload
• occasional throttling

ProxiesAPI helps with:

• request distribution
• cleaner network paths
• scaling repeated jobs

You usually want both:

• sane retry logic first
• better network plumbing second

If the retry policy is bad, adding proxies just helps you fail in more places.

Final thoughts

Good retry logic is not about maximizing attempts. It is about maximizing useful attempts.

For most scrapers, the winning pattern is:

1. retry only transient failures
2. use exponential backoff
3. add jitter
4. respect Retry-After
5. stop when the site is clearly blocking you

That gives you a scraper that recovers from real noise without creating its own problems.