Amazon Best Sellers Scraper: Track Category Rankings and Price Moves

Jul 11, 2026 · tutorial · #amazon best sellers scraper, #amazon, #python, #price tracking, #web-scraping, #beautifulsoup, #proxiesapi

If you want to monitor product momentum on Amazon, the Best Sellers pages are one of the highest-signal starting points. They already expose the ranking order for a category, and many pages also show enough pricing and review data to turn a daily snapshot into a real trend dataset.

The important part is being honest: Amazon can throttle, serve robot checks, or vary the markup. A useful scraper therefore needs two layers:

a fetch wrapper that detects blocks early
a parser that relies on the repeated Best Sellers card structure

What we are scraping

The root Best Sellers page is:

https://www.amazon.com/Best-Sellers/zgbs

Subcategories hang off the left navigation tree, for example:

https://www.amazon.com/Best-Sellers-Kitchen-Dining/zgbs/kitchen

On the current HTML:

rank badges appear in span.zg-bdg-text
each product card is div.p13n-sc-uncoverable-faceout
product titles appear in div.p13n-sc-truncate
ratings are exposed through span.a-icon-alt
review counts often appear in div.a-icon-row span.a-size-small

That is enough to build a best-sellers tracker without pretending the page is static forever.

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml pandas

Optional:

export PROXIESAPI_PROXY_URL="http://USER:PASS@gateway.proxiesapi.com:PORT"

Step 1: Build a fetch wrapper that detects Amazon block pages

from __future__ import annotations

import os
import random
import re
import time
from dataclasses import dataclass

import requests

TIMEOUT = (10, 30)
PROXIESAPI_PROXY_URL = os.getenv("PROXIESAPI_PROXY_URL", "").strip()

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
]


@dataclass
class FetchResult:
    url: str
    status_code: int
    text: str


def proxy_dict() -> dict | None:
    if not PROXIESAPI_PROXY_URL:
        return None
    return {"http": PROXIESAPI_PROXY_URL, "https": PROXIESAPI_PROXY_URL}


def looks_blocked(html: str) -> bool:
    if not html:
        return True
    needles = [
        "Robot Check",
        "validateCaptcha",
        "Sorry, we just need to make sure you're not a robot",
    ]
    lowered = html.lower()
    return any(token.lower() in lowered for token in needles)


def fetch(session: requests.Session, url: str, max_retries: int = 4) -> FetchResult:
    last_exc = None

    for attempt in range(1, max_retries + 1):
        try:
            headers = {
                "User-Agent": random.choice(USER_AGENTS),
                "Accept-Language": "en-US,en;q=0.9",
                "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            }
            response = session.get(url, headers=headers, timeout=TIMEOUT, proxies=proxy_dict())
            text = response.text or ""

            if response.status_code in {429, 503} or looks_blocked(text):
                raise RuntimeError(f"blocked status={response.status_code}")

            response.raise_for_status()
            return FetchResult(url=url, status_code=response.status_code, text=text)
        except Exception as exc:
            last_exc = exc
            time.sleep(min(12, 1.7 ** attempt) + random.random())

    raise RuntimeError(f"failed to fetch {url}: {last_exc}")

This wrapper does not promise magic bypasses. It just makes failures explicit instead of silently parsing garbage.

Step 2: Parse Best Sellers product cards

from urllib.parse import urljoin

from bs4 import BeautifulSoup

AMAZON_BASE = "https://www.amazon.com"


def parse_price(text: str | None) -> float | None:
    if not text:
        return None
    match = re.search(r"([0-9]+(?:\.[0-9]{2})?)", text.replace(",", ""))
    return float(match.group(1)) if match else None


def parse_rating(text: str | None) -> float | None:
    if not text:
        return None
    match = re.search(r"(\d+(?:\.\d+)?)\s*out of\s*5", text)
    return float(match.group(1)) if match else None


def parse_int(text: str | None) -> int | None:
    if not text:
        return None
    match = re.search(r"(\d[\d,]*)", text)
    return int(match.group(1).replace(",", "")) if match else None


def parse_best_sellers_page(html: str, category: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")

    ranks = [node.get_text(strip=True) for node in soup.select("span.zg-bdg-text")]
    cards = soup.select("div.p13n-sc-uncoverable-faceout")

    rows = []
    for idx, card in enumerate(cards):
        title_node = card.select_one("div.p13n-sc-truncate")
        link = card.select_one('a.a-link-normal.aok-block[href*="/dp/"]')
        rating_node = card.select_one("span.a-icon-alt")
        review_node = card.select_one("div.a-icon-row span.a-size-small")

        price = None
        for span in card.select("span"):
            candidate = span.get_text(" ", strip=True)
            if candidate.startswith("$") and parse_price(candidate) is not None:
                price = parse_price(candidate)
                break

        product_url = urljoin(AMAZON_BASE, link.get("href")) if link else None
        asin = card.get("id")

        rows.append(
            {
                "category": category,
                "rank": parse_int(ranks[idx]) if idx < len(ranks) else idx + 1,
                "asin": asin,
                "title": title_node.get_text(" ", strip=True) if title_node else None,
                "product_url": product_url,
                "rating": parse_rating(rating_node.get_text(" ", strip=True) if rating_node else None),
                "review_count": parse_int(review_node.get_text(" ", strip=True) if review_node else None),
                "price": price,
            }
        )

    return rows

Step 3: Save daily snapshots and compute movement

The real value is not a single scrape. It is the delta between snapshots.

from pathlib import Path

import pandas as pd


def write_snapshot(rows: list[dict], snapshot_date: str, out_dir: str = "out") -> Path:
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    path = out / f"amazon_best_sellers_{snapshot_date}.csv"
    pd.DataFrame(rows).to_csv(path, index=False)
    return path


def compare_snapshots(previous_csv: str, current_csv: str) -> pd.DataFrame:
    previous = pd.read_csv(previous_csv).rename(
        columns={"rank": "rank_prev", "price": "price_prev"}
    )
    current = pd.read_csv(current_csv).rename(
        columns={"rank": "rank_now", "price": "price_now"}
    )

    merged = current.merge(previous[["asin", "rank_prev", "price_prev"]], on="asin", how="left")
    merged["rank_delta"] = merged["rank_prev"] - merged["rank_now"]
    merged["price_delta"] = merged["price_now"] - merged["price_prev"]
    return merged.sort_values(["rank_now"])

Runner:

if __name__ == "__main__":
    session = requests.Session()
    result = fetch(session, "https://www.amazon.com/Best-Sellers/zgbs")
    rows = parse_best_sellers_page(result.text, category="Best Sellers")
    current = write_snapshot(rows, snapshot_date="2026-07-11")
    print(rows[:3])

What makes this useful in practice

A best-sellers scraper is not just a list of products. It becomes useful when you track:

rank gainers
products with sudden review acceleration
price drops attached to rank jumps
category leaders that remain sticky for weeks

That is why a snapshot schema matters more than clever parsing tricks.

Field	Why keep it
`asin`	stable product key
`rank`	direct category position
`price`	best-effort pricing signal
`rating`	quick quality proxy
`review_count`	social proof growth

Important limitations

Amazon markup changes often enough that you should expect selector maintenance
some categories expose better price data than others
block pages can look like valid HTML unless you check for them explicitly
scraping Amazon may violate Terms of Service depending on your use case

That is why the safest posture is:

detect blocks early
persist raw HTML samples when a parser fails
use ProxiesAPI or another proxy layer only as part of a broader reliability strategy

If your goal is tracking bestseller movement over time, this pattern gives you a realistic foundation without pretending Amazon scraping is frictionless.

Use ProxiesAPI when Best Sellers pages become unreliable

Amazon is aggressive about bot detection. ProxiesAPI will not make scraping risk disappear, but it gives you a cleaner proxy layer so retries, rotation, and block detection are easier to manage.

Get 1,000 free API calls View pricing

Build a practical IMDb scraper that starts from the search suggestion endpoint, enriches title pages, and exports ratings, cast, and release dates with a ProxiesAPI-ready fetch layer.

tutorial#python#imdb scraper#imdb

Scrape Hacker News Jobs Posts with Python + ProxiesAPI

Turn the HN Jobs feed into a clean dataset of roles, companies, domains, and links from the real jobs page with resilient pagination and a validation screenshot.

tutorial#python#hackernews#jobs

Scrape Numbeo Rent Prices and Cost Breakdown by City

Extract Numbeo city tables for rent and living-cost items, normalize ranges, and build a practical city comparison dataset with a validation screenshot.

tutorial#python#numbeo#rent

Scrape Crunchbase Company Data

Collect company profile fields from Crunchbase by discovering organization URLs, rendering profile pages, and parsing structured data into CSV.

tutorial#python#crunchbase#web-scraping