Web Scraping with HTTPX: Async Fetching, Retries, and Timeouts
Web scraping with HTTPX is a good middle ground between old-school requests scripts and a full async crawling stack.
You get:
- a clean sync API
- an
AsyncClientwhen you need concurrency - HTTP/2 support
- connection pooling
- explicit timeout controls
What you do not get is a free pass on scraper hygiene. Fast clients can still fail fast if you skip retries, timeouts, and pacing.
This guide shows a practical template for web scraping with HTTPX that you can actually reuse in production.
HTTPX gives you a modern client and great async ergonomics. When the real problem becomes bans, burst failures, or IP reputation, ProxiesAPI is the clean next layer to add without rewriting the scraper.
Why teams reach for HTTPX
For Python scraping work, HTTPX is appealing because one library covers both sync and async usage.
That means you can:
- start with a simple synchronous scraper
- move to async fetching later
- keep nearly the same calling style
The other useful detail is that HTTPX includes timeouts by default. According to the official docs, the default inactivity timeout is five seconds, which is much safer than leaving sockets hanging forever.
When HTTPX is a better fit than requests
| Tool | Best when | Strength | Tradeoff |
|---|---|---|---|
requests | small sync jobs | simplest mental model | no native async client |
httpx | growing scrapers that may need async | one API for sync + async, better timeout model | you still design retries yourself |
aiohttp | very custom async crawlers | mature async ecosystem | more boilerplate if you also need sync flows |
If you already know you want concurrency, HTTPX is often the least awkward upgrade path.
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install httpx beautifulsoup4 lxml
Step 1: Use explicit timeouts and connection limits
Do not rely on defaults once the scraper matters.
from __future__ import annotations
import httpx
TIMEOUT = httpx.Timeout(connect=10.0, read=30.0, write=30.0, pool=30.0)
LIMITS = httpx.Limits(max_connections=20, max_keepalive_connections=10)
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def build_client() -> httpx.Client:
return httpx.Client(
timeout=TIMEOUT,
limits=LIMITS,
headers=HEADERS,
follow_redirects=True,
http2=True,
)
Why this matters:
- connect timeout keeps dead hosts from hanging forever
- read timeout protects you from slow upstreams
- connection limits stop your own client from stampeding
Step 2: Add explicit retries
Retry policy is part of scraper behavior, so keep it visible in your own code.
import random
import time
RETRY_STATUS_CODES = {429, 500, 502, 503, 504}
def fetch_html(client: httpx.Client, url: str, *, max_attempts: int = 4) -> str:
last_error = None
for attempt in range(1, max_attempts + 1):
try:
response = client.get(url)
if response.status_code in RETRY_STATUS_CODES:
raise httpx.HTTPStatusError(
f"retryable status {response.status_code}",
request=response.request,
response=response,
)
response.raise_for_status()
return response.text
except (httpx.TimeoutException, httpx.NetworkError, httpx.HTTPStatusError) as exc:
last_error = exc
sleep_s = min(20.0, 2 ** attempt) + random.random()
time.sleep(sleep_s)
raise RuntimeError(f"failed to fetch {url}: {last_error}")
This is intentionally conservative. For scraping, boring retry behavior beats clever retry behavior.
Step 3: Parse a page cleanly
HTTPX only solves the network side. You still want a separate parser.
from bs4 import BeautifulSoup
def parse_page(html: str, url: str) -> dict:
soup = BeautifulSoup(html, "lxml")
title = soup.title.get_text(" ", strip=True) if soup.title else None
h1 = soup.select_one("h1")
h1_text = h1.get_text(" ", strip=True) if h1 else None
return {
"url": url,
"title": title,
"h1": h1_text,
"html_len": len(html),
}
The clean split is:
- HTTPX fetches
- BeautifulSoup parses
- your exporter writes rows
That separation is what keeps scraper maintenance sane.
Step 4: Move to async fetching without rewriting everything
This is the main reason people choose web scraping with HTTPX in the first place.
import asyncio
async def fetch_html_async(
client: httpx.AsyncClient,
url: str,
*,
sem: asyncio.Semaphore,
max_attempts: int = 4,
) -> str:
last_error = None
async with sem:
for attempt in range(1, max_attempts + 1):
try:
response = await client.get(url)
if response.status_code in RETRY_STATUS_CODES:
raise httpx.HTTPStatusError(
f"retryable status {response.status_code}",
request=response.request,
response=response,
)
response.raise_for_status()
return response.text
except (httpx.TimeoutException, httpx.NetworkError, httpx.HTTPStatusError) as exc:
last_error = exc
await asyncio.sleep(min(20.0, 2 ** attempt) + random.random())
raise RuntimeError(f"failed to fetch {url}: {last_error}")
async def crawl(urls: list[str]) -> list[dict]:
sem = asyncio.Semaphore(10)
async with httpx.AsyncClient(
timeout=TIMEOUT,
limits=LIMITS,
headers=HEADERS,
follow_redirects=True,
http2=True,
) as client:
pages = await asyncio.gather(
*(fetch_html_async(client, url, sem=sem) for url in urls),
return_exceptions=True,
)
out = []
for url, result in zip(urls, pages):
if isinstance(result, Exception):
out.append({"url": url, "ok": False, "error": str(result)})
else:
out.append({"url": url, "ok": True, **parse_page(result, url)})
return out
Two important details:
- concurrency is bounded with a semaphore
- the parser stays unchanged
That is the real productivity win.
Common mistakes when scraping with HTTPX
| Mistake | What happens | Better move |
|---|---|---|
asyncio.gather() with hundreds of URLs at once | you spike traffic and trigger bans | use a semaphore and small batches |
| no explicit timeout config | sockets hang and jobs stall | set connect/read/pool timeouts |
| retrying every failure forever | you hammer a struggling site | cap attempts and back off |
| mixing fetch and parse logic | debugging becomes painful | keep network and parser separate |
HTTPX is fast enough that these mistakes become visible quickly.
Where ProxiesAPI fits
HTTPX solves request ergonomics. It does not solve IP reputation, rate limits, or traffic distribution.
That is where a proxy layer can help:
- keep a clean request interface
- rotate traffic when scale grows
- reduce the blast radius of one noisy run
A good rule is:
- first fix timeouts
- then fix retries
- then fix concurrency
- only then add ProxiesAPI if the failure mode is still network-side blocking
Wrap-up
Web scraping with HTTPX works well because it keeps the upgrade path simple:
- sync first
- async later
- same general API
If you pair that with explicit timeouts, bounded concurrency, and honest retry behavior, you get a scraper that is fast without turning into a support ticket generator.
HTTPX gives you a modern client and great async ergonomics. When the real problem becomes bans, burst failures, or IP reputation, ProxiesAPI is the clean next layer to add without rewriting the scraper.