Web Scraping Rate Limiting: How to Throttle Requests Without Killing Throughput

Most rate-limiting advice for scraping is too vague to be useful. It says things like "slow down your requests" without answering the real question:

How do you slow down enough to avoid bans without slowing down so much that the crawl becomes useless?

The answer is not one fixed requests-per-second number. Good scraper throttling balances four things at once:

• target sensitivity
• request cost per page
• crawl deadline
• failure feedback from the site

When you get that balance right, you stop treating rate limiting as punishment and start using it as a throughput control system.

The three levers that actually matter

If you only change one of these, you usually get unstable results. Good throttling coordinates all three.

Start with a token bucket, not random sleep()

Random sleeps are fine for tiny scripts, but they break down fast. A token bucket is a better default because it gives you a steady request rate while still allowing small bursts.

import asyncio
import time


class TokenBucket:
    def __init__(self, rate_per_sec: float, capacity: int):
        self.rate = rate_per_sec
        self.capacity = capacity
        self.tokens = capacity
        self.updated_at = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self) -> None:
        while True:
            async with self.lock:
                now = time.monotonic()
                elapsed = now - self.updated_at
                self.updated_at = now
                self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)

                if self.tokens >= 1:
                    self.tokens -= 1
                    return

                wait_for = (1 - self.tokens) / self.rate
            await asyncio.sleep(wait_for)

This gives you:

• a controlled average rate
• short bursts up to capacity
• one place to tune behavior

That is much cleaner than sprinkling await asyncio.sleep(2) throughout your crawler.

Combine rate limits with concurrency limits

Request rate alone is not enough. A site can still get slammed if each page is slow and you let too many requests pile up at once.

import aiohttp
import asyncio

bucket = TokenBucket(rate_per_sec=1.5, capacity=3)
semaphore = asyncio.Semaphore(5)


async def fetch(session: aiohttp.ClientSession, url: str) -> str:
    await bucket.acquire()
    async with semaphore:
        async with session.get(url, timeout=aiohttp.ClientTimeout(total=45)) as resp:
            resp.raise_for_status()
            return await resp.text()

This pattern solves two different problems:

• the bucket smooths how often work starts
• the semaphore caps how much work is happening at once

Use both.

Back off harder on signals that matter

Not every failure should trigger the same retry plan.

Here's a simple retry wrapper:

import random


async def fetch_with_backoff(session, url, attempts=5):
    delay = 2.0
    for attempt in range(1, attempts + 1):
        try:
            return await fetch(session, url)
        except aiohttp.ClientResponseError as exc:
            if exc.status == 429:
                await asyncio.sleep(delay + random.uniform(0.0, 0.5))
                delay = min(delay * 2, 60)
                continue
            if exc.status == 403:
                await asyncio.sleep(delay * 2)
                delay = min(delay * 2, 90)
                continue
            raise
        except Exception:
            if attempt == attempts:
                raise
            await asyncio.sleep(delay + random.uniform(0.0, 0.5))
            delay = min(delay * 2, 30)

The important idea is adaptive response. A 429 should teach the scraper to slow down, not just mindlessly retry at the same speed.

Per-domain limits beat one global limit

If your crawler hits multiple domains, one global throttle is usually wrong.

Example:

• a simple content site may tolerate 2 req/s
• a fragile marketplace search page may only tolerate 0.3 req/s
• an internal API endpoint might allow much more

Instead of one global bucket, keep one bucket per host or per route family.

buckets = {
    "docs.example.com": TokenBucket(rate_per_sec=2.0, capacity=4),
    "shop.example.com": TokenBucket(rate_per_sec=0.4, capacity=1),
}

That single change often improves throughput more than any random "stealth" tweak, because you stop over-throttling easy targets and under-throttling sensitive ones.

The throughput math people forget

Throttle design should start with a rough budget:

pages_to_crawl / acceptable_runtime_seconds = required_pages_per_second

If you need 18,000 pages in 6 hours, your floor is roughly:

18000 / 21600 = 0.83 pages/sec

That tells you the system cannot survive at 0.1 req/s no matter how "safe" that sounds. Instead, you need a smarter design:

• lower-cost pages first
• per-domain buckets
• queues
• retry caps
• a proxy/network layer that stays stable under load

This is where tooling such as ProxiesAPI helps. It does not replace throttling, but it reduces the number of transport failures your rate limiter has to absorb.

A practical starting point

If you have no benchmark data yet, start here:

Then tune with actual telemetry:

• success rate
• median response time
• 429 frequency
• queue backlog

If you are not measuring those, you are guessing.

The goal is steady throughput

The best scraper is not the one that sprints. It's the one that finishes.

Good rate limiting gives you:

• fewer bans
• fewer retry storms
• more predictable completion times
• cleaner use of proxies and browser sessions

That is why "slow enough to survive" and "fast enough to matter" are not opposites. They are the same engineering problem.