Scrape Book Data from Goodreads
Goodreads is one of the richest public web datasets for books: titles, authors, rating counts, review counts, series links, and publication metadata are all visible on public pages.
In this tutorial we will build a real Python scraper that:
- starts from a Goodreads list page such as
Best Books Ever - extracts book URLs
- visits each book page and extracts title, author, average rating, rating count, and review count
- exports JSON and CSV
- keeps ProxiesAPI isolated in the network layer
Make Goodreads crawls resilient with ProxiesAPI
Goodreads pages are big and requests add up fast when you scrape lists to books to reviews. ProxiesAPI belongs in your fetch layer so you can add retries and routing without changing your parser.
What we are scraping
The list page we will use:
https://www.goodreads.com/list/show/1.Best_Books_Ever?page=1
On the live page, each row includes:
- a link to
/book/show/... - the visible book title
- the visible author link
- average rating text such as
4.35 avg rating - rating count text such as
10,174,862 ratings
That gives us a stable two-step pipeline:
- scrape book URLs from the list page
- visit each book page for richer metadata
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml pandas
Optional ProxiesAPI key:
export PROXIESAPI_KEY="YOUR_PROXIESAPI_KEY"
Step 1: Fetch HTML with retries
from __future__ import annotations
import os
import random
import time
import urllib.parse
import requests
PROXIESAPI_KEY = os.getenv("PROXIESAPI_KEY", "")
TIMEOUT = (10, 45)
session = requests.Session()
session.headers.update(
{
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/125.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
)
def proxiesapi_url(target_url: str) -> str:
if not PROXIESAPI_KEY:
return target_url
return (
"https://api.proxiesapi.com/?auth_key="
+ urllib.parse.quote(PROXIESAPI_KEY, safe="")
+ "&url="
+ urllib.parse.quote(target_url, safe="")
)
def fetch(url: str, max_retries: int = 4) -> str:
last_error = None
for attempt in range(1, max_retries + 1):
try:
r = session.get(proxiesapi_url(url), timeout=TIMEOUT)
r.raise_for_status()
if len(r.text) < 3000:
raise RuntimeError(f"suspiciously short HTML ({len(r.text)} bytes)")
return r.text
except Exception as exc:
last_error = exc
time.sleep(min(12, 2 ** (attempt - 1)) + random.random())
raise RuntimeError(f"fetch failed after {max_retries} attempts: {last_error}")
The important design choice: ProxiesAPI affects only the requested URL, not the parsing code.
Step 2: Extract book URLs from the list page
Goodreads list pages contain many anchors where href includes /book/show/.
from bs4 import BeautifulSoup
from urllib.parse import urljoin
BASE = "https://www.goodreads.com"
def extract_book_urls(list_html: str) -> list[str]:
soup = BeautifulSoup(list_html, "lxml")
seen: set[str] = set()
out: list[str] = []
for a in soup.select('a[href*="/book/show/"]'):
href = a.get("href")
if not href:
continue
abs_url = urljoin(BASE, href).split("?")[0].split("#")[0]
if abs_url in seen:
continue
seen.add(abs_url)
out.append(abs_url)
return out
list_html = fetch("https://www.goodreads.com/list/show/1.Best_Books_Ever?page=1")
book_urls = extract_book_urls(list_html)
print("books found:", len(book_urls))
print(book_urls[:5])
This dedupe step matters because Goodreads often repeats book links within the same row.
Step 3: Parse a Goodreads book page
We will collect:
- title
- author
- average rating
- rating count
- review count
- publication year when available
import json
import re
def clean_text(s: str | None) -> str | None:
if not s:
return None
return re.sub(r"\s+", " ", s).strip() or None
def parse_int(text: str | None) -> int | None:
if not text:
return None
m = re.search(r"(\d[\d,]*)", text)
return int(m.group(1).replace(",", "")) if m else None
def parse_book(html: str, url: str) -> dict:
soup = BeautifulSoup(html, "lxml")
title = clean_text(soup.select_one("h1").get_text(" ", strip=True) if soup.select_one("h1") else None)
author = None
author_a = soup.select_one('a[href*="/author/show/"]')
if author_a:
author = clean_text(author_a.get_text(" ", strip=True))
rating_value = None
rating_count = None
review_count = None
for tag in soup.select('script[type="application/ld+json"]'):
raw = tag.get_text(strip=True)
if not raw:
continue
try:
data = json.loads(raw)
except Exception:
continue
items = data if isinstance(data, list) else [data]
for item in items:
if not isinstance(item, dict):
continue
agg = item.get("aggregateRating")
if isinstance(agg, dict):
rating_value = agg.get("ratingValue") or rating_value
rating_count = parse_int(str(agg.get("ratingCount"))) or rating_count
review_count = parse_int(str(agg.get("reviewCount"))) or review_count
if not title and item.get("name"):
title = clean_text(item.get("name"))
if rating_count is None or review_count is None:
for el in soup.select("span, div, a"):
text = el.get_text(" ", strip=True)
low = text.lower()
if "ratings" in low and rating_count is None:
rating_count = parse_int(text)
if "reviews" in low and review_count is None:
review_count = parse_int(text)
pub_year = None
blob = soup.get_text(" ", strip=True)
m = re.search(r"Published\s+\w+\s+\d{1,2},\s+(\d{4})", blob)
if m:
pub_year = int(m.group(1))
return {
"url": url,
"title": title,
"author": author,
"average_rating": rating_value,
"rating_count": rating_count,
"review_count": review_count,
"publication_year": pub_year,
}
This parser is deliberately conservative: JSON-LD first, visible text second.
Step 4: Crawl a list of books
def scrape_list(list_url: str, limit: int = 25, pause_seconds: float = 1.0) -> list[dict]:
list_html = fetch(list_url)
book_urls = extract_book_urls(list_html)
rows: list[dict] = []
for i, url in enumerate(book_urls[:limit], start=1):
html = fetch(url)
row = parse_book(html, url)
rows.append(row)
print(f"{i}/{min(limit, len(book_urls))} {row['title']}")
time.sleep(pause_seconds)
return rows
If you need review text or series expansion later, keep that as a second-stage enrichment job instead of mixing it into the first crawl.
Export JSON and CSV
import json
import pandas as pd
if __name__ == "__main__":
start_url = "https://www.goodreads.com/list/show/1.Best_Books_Ever?page=1"
rows = scrape_list(start_url, limit=20)
with open("goodreads-books.json", "w", encoding="utf-8") as fh:
json.dump(rows, fh, ensure_ascii=False, indent=2)
pd.DataFrame(rows).to_csv("goodreads-books.csv", index=False)
print("saved", len(rows), "books")
Practical advice
- Keep the first crawl narrow. List page to book page is enough for many catalog jobs.
- Log page titles when parsing fails. That helps detect consent screens or soft blocks quickly.
- Sleep between requests. Goodreads pages are heavy, and politeness reduces random failures.
- Treat review crawling as a separate stage. The request count climbs much faster once you add review pages.
This pattern is enough to build a clean Goodreads dataset with titles, authors, ratings, and review counts while keeping the network layer simple.
Make Goodreads crawls resilient with ProxiesAPI
Goodreads pages are big and requests add up fast when you scrape lists to books to reviews. ProxiesAPI belongs in your fetch layer so you can add retries and routing without changing your parser.
Related guides
Scrape Book Data from Goodreads (Titles, Authors, Ratings, and Reviews)
A practical Goodreads scraper in Python: collect book title/author/rating count/review count + key metadata using robust selectors, ProxiesAPI in the fetch layer, and export to JSON/CSV.
tutorial#python#goodreads#books
Scrape Book Reviews and Ratings from Goodreads
Extract Goodreads book metadata, average rating, rating counts, review counts, and top review snippets with Python using JSON-LD plus __NEXT_DATA__ review objects.
tutorial#python#goodreads#books
Scrape Book Data from Goodreads with Python (List Pages + Pagination)
Scrape Goodreads list pages for title/author/rating/reviews with Python: fetch via ProxiesAPI, parse real HTML selectors, paginate safely, and export CSV/JSON.
tutorial#python#goodreads#books
Scrape GitHub Trending Repositories with Python
Build a daily GitHub Trending dataset with Python: collect repository names, languages, star counts, and URLs, then export clean CSV or JSON with an optional ProxiesAPI fetch layer.
tutorial#python#github#web-scraping