Requests vs HTTPX for Web Scraping: Sync, Async, Retries, and Throughput

If you scrape in Python, you will eventually hit this decision:

Should this scraper stay on requests, or is it time to move to httpx?

That question matters because the right answer changes with workload.

For a small scraper, requests is often the best choice because it is:

• familiar
• boring
• readable
• easy to debug

For a growing scraper, httpx becomes attractive because it gives you:

• both sync and async APIs
• better upgrade path to concurrency
• cleaner timeout model
• HTTP/2 support where it helps

The mistake is treating this as ideology. It is really an operations tradeoff.

The short version

If you are scraping five pages every hour, use requests.

If you are scraping thousands of pages with bounded concurrency, httpx is usually the more future-proof bet.

Why requests is still hard to beat

The biggest advantage of requests is not performance. It is clarity.

import requests

response = requests.get(
    "https://example.com",
    headers={"User-Agent": "Mozilla/5.0"},
    timeout=(10, 30),
)
response.raise_for_status()
html = response.text

Everyone on the team understands what that does.

For web scraping, that matters because most failures are operational:

• wrong target URL
• missing headers
• selector drift
• bad retry policy
• site-level blocking

When the fetch code is simple, you debug the real problem faster.

Use requests when:

• the scraper is synchronous
• throughput is modest
• maintainability beats raw concurrency
• the team already has a stable retry pattern

Where HTTPX starts to win

HTTPX gives you a sync client and an async client with a very similar API.

That is a huge advantage when a scraper outgrows its first version.

import httpx

timeout = httpx.Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
limits = httpx.Limits(max_connections=20, max_keepalive_connections=10)

with httpx.Client(timeout=timeout, limits=limits, headers={"User-Agent": "Mozilla/5.0"}) as client:
    response = client.get("https://example.com", follow_redirects=True)
    response.raise_for_status()
    html = response.text

Then later:

import asyncio
import httpx


async def fetch_many(urls: list[str]) -> list[str]:
    timeout = httpx.Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
    limits = httpx.Limits(max_connections=20, max_keepalive_connections=10)
    sem = asyncio.Semaphore(10)

    async with httpx.AsyncClient(
        timeout=timeout,
        limits=limits,
        headers={"User-Agent": "Mozilla/5.0"},
        follow_redirects=True,
        http2=True,
    ) as client:
        async def one(url: str) -> str:
            async with sem:
                response = await client.get(url)
                response.raise_for_status()
                return response.text

        return await asyncio.gather(*(one(url) for url in urls))

That is the main reason scraper teams switch. They do not switch because requests suddenly became bad. They switch because concurrency became a first-class requirement.

Retries: neither library saves you automatically

This part is important.

People often compare the libraries as if one of them gives "production retries" out of the box. In practice, both need explicit scraper-level retry design.

With requests:

import random
import time
import requests

RETRYABLE = {429, 500, 502, 503, 504}


def fetch_with_requests(url: str, attempts: int = 4) -> str:
    last_error = None
    for attempt in range(1, attempts + 1):
        try:
            r = requests.get(url, timeout=(10, 30), headers={"User-Agent": "Mozilla/5.0"})
            if r.status_code in RETRYABLE:
                raise RuntimeError(f"retryable status {r.status_code}")
            r.raise_for_status()
            return r.text
        except Exception as exc:
            last_error = exc
            if attempt == attempts:
                break
            time.sleep(min(12, 2 ** attempt) + random.random())
    raise RuntimeError(f"failed: {last_error}")

With HTTPX:

import random
import time
import httpx

RETRYABLE = {429, 500, 502, 503, 504}


def fetch_with_httpx(url: str, attempts: int = 4) -> str:
    timeout = httpx.Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)

    with httpx.Client(timeout=timeout, headers={"User-Agent": "Mozilla/5.0"}) as client:
        last_error = None
        for attempt in range(1, attempts + 1):
            try:
                r = client.get(url, follow_redirects=True)
                if r.status_code in RETRYABLE:
                    raise RuntimeError(f"retryable status {r.status_code}")
                r.raise_for_status()
                return r.text
            except Exception as exc:
                last_error = exc
                if attempt == attempts:
                    break
                time.sleep(min(12, 2 ** attempt) + random.random())
        raise RuntimeError(f"failed: {last_error}")

The lesson is simple: the library matters less than your retry policy.

Sync vs async in real scraping work

Async does not automatically mean faster results in production.

It helps when:

• the target is network-bound
• you have many independent URLs
• you cap concurrency responsibly
• the site tolerates parallel fetching

It does not help much when:

• parsing dominates runtime
• the site blocks aggressive bursts
• you are only fetching a handful of pages
• your real bottleneck is IP reputation

This is why many teams do perfectly well with requests longer than they expect.

Throughput tradeoffs that actually matter

The client choice should follow the crawl shape, not internet fashion.

What about proxies?

Both libraries handle proxies fine enough for most scraping workloads.

With requests, you commonly pass a proxies dict. With HTTPX, you configure proxy behavior through the client. In both cases, the higher-order decision is the same:

• is the target fetchable at all?
• are bans or burst failures your real problem?
• do you need a managed proxy layer yet?

That is where ProxiesAPI fits in. It is not a substitute for good client design. It is the next layer once the scraper's network reliability starts failing for reasons your HTTP client alone cannot solve.

My recommendation

Start with requests if:

• you are building the first version
• the job is synchronous
• clarity is more important than theoretical scalability

Start with HTTPX if:

• you already know concurrency is part of the requirement
• you want one library for sync now and async later
• you expect connection limits and timeout tuning to matter early

Do not migrate just to migrate.

Migrate when the workload tells you to.

Final takeaway

For requests vs httpx for web scraping, the honest answer is:

• requests wins on simplicity
• HTTPX wins on growth path
• neither one replaces disciplined retries, pacing, and parser hygiene

Pick the client that matches the actual crawl you are running today, then add ProxiesAPI when the network layer becomes the bottleneck instead of the Python code.