Scrape Stack Overflow Questions and Answers

Jun 28, 2026 · tutorial · #python, #stack-overflow, #web-scraping, #requests, #beautifulsoup, #lxml, #json, #proxies

Stack Overflow is a strong scraping target because the HTML is still readable without a headless browser, the URLs are predictable, and the accepted-answer pattern is obvious enough to parse reliably.

In this guide we will:

crawl a tag page
collect question URLs and summary stats
fetch each question page
extract the title, score, tags, accepted answer, and code blocks

The result is useful for internal search, dataset building, or support knowledge bases.

Stack Overflow tag page

Scale Stack Overflow crawls without rebuilding your network layer

Stack Overflow is easy to scrape in small batches. When you fan out across tags and thousands of question pages, ProxiesAPI helps keep retries, IP reputation, and fetch consistency under control.

Get 1,000 free API calls View pricing

URLs and page structure

Two page types matter:

tag listing: https://stackoverflow.com/questions/tagged/python?tab=Newest&page=1&pagesize=50
question detail: https://stackoverflow.com/questions/<id>/<slug>

On the tag page, question cards are typically rendered as div.s-post-summary. Inside each card you will usually find:

h3 a.s-link for the title and URL
stat blocks for votes, answers, and views

On the detail page, the selectors worth anchoring to are:

h1 a.question-hyperlink or h1
div.question div.s-prose
div.answer
div.js-vote-count
time[itemprop='dateCreated']

Accepted answers are usually marked by the accepted-answer class or an accepted-answer indicator inside the answer block.

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml python-dotenv

.env:

PROXIESAPI_KEY="YOUR_PROXIESAPI_KEY"

Step 1: Reusable fetcher with timeout, backoff, and optional proxy routing

import os
import random
import time
from urllib.parse import quote

import requests
from dotenv import load_dotenv

load_dotenv()

BASE = "https://stackoverflow.com"
PROXIESAPI_KEY = os.getenv("PROXIESAPI_KEY", "").strip()
TIMEOUT = (10, 30)

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}


class HttpClient:
    def __init__(self) -> None:
        self.session = requests.Session()
        self.session.headers.update(HEADERS)

    def _wrap_url(self, target_url: str) -> str:
        if not PROXIESAPI_KEY:
            return target_url
        return f"https://api.proxiesapi.com/?auth_key={PROXIESAPI_KEY}&url={quote(target_url, safe='')}"

    def get_html(self, target_url: str, retries: int = 4) -> str:
        last_error = None

        for attempt in range(1, retries + 1):
            try:
                response = self.session.get(self._wrap_url(target_url), timeout=TIMEOUT)
                response.raise_for_status()
                text = response.text

                if "captcha" in text.lower() and "stack overflow" in text.lower():
                    raise RuntimeError("captcha or challenge page detected")

                return text
            except Exception as exc:
                last_error = exc
                time.sleep(min(2 ** attempt, 8) + random.random())

        raise RuntimeError(f"failed to fetch {target_url}: {last_error}")

Step 2: Parse a tag page

import re
from urllib.parse import urljoin

from bs4 import BeautifulSoup


def parse_int(text: str) -> int | None:
    match = re.search(r"(\d[\d,]*)", text or "")
    if not match:
        return None
    return int(match.group(1).replace(",", ""))


def parse_tag_page(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    rows = []

    for card in soup.select("div.s-post-summary"):
        title_link = card.select_one("h3 a.s-link")
        if not title_link:
            continue

        stats = card.select("span.s-post-summary--stats-item-number")
        votes = parse_int(stats[0].get_text(" ", strip=True)) if len(stats) > 0 else None
        answers = parse_int(stats[1].get_text(" ", strip=True)) if len(stats) > 1 else None
        views = parse_int(stats[2].get_text(" ", strip=True)) if len(stats) > 2 else None

        tag_names = [tag.get_text(strip=True) for tag in card.select("a.post-tag")]

        rows.append(
            {
                "title": title_link.get_text(" ", strip=True),
                "url": urljoin(BASE, title_link.get("href", "")),
                "votes": votes,
                "answers": answers,
                "views": views,
                "tags": tag_names,
            }
        )

    return rows

Usage:

client = HttpClient()
html = client.get_html("https://stackoverflow.com/questions/tagged/python?tab=Newest&page=1&pagesize=15")
questions = parse_tag_page(html)
print("questions:", len(questions))
print(questions[0])

Step 3: Parse a question page and detect the accepted answer

def text_or_none(node) -> str | None:
    return node.get_text("\n", strip=True) if node else None


def parse_question_page(html: str) -> dict:
    soup = BeautifulSoup(html, "lxml")

    title_el = soup.select_one("h1 a.question-hyperlink") or soup.select_one("h1")
    question_block = soup.select_one("div.question")
    question_body = question_block.select_one("div.s-prose") if question_block else None
    vote_el = question_block.select_one("div.js-vote-count") if question_block else None
    time_el = soup.select_one("time[itemprop='dateCreated']") or soup.select_one("time")

    accepted_answer = None
    for answer in soup.select("div.answer"):
        classes = answer.get("class", [])
        if "accepted-answer" in classes or answer.select_one(".js-accepted-answer-indicator"):
            accepted_answer = answer
            break

    accepted_body = accepted_answer.select_one("div.s-prose") if accepted_answer else None

    code_blocks = []
    for code in soup.select("div.question div.s-prose pre code, div.answer div.s-prose pre code"):
        snippet = code.get_text("\n", strip=True)
        if snippet:
            code_blocks.append(snippet)

    return {
        "title": text_or_none(title_el),
        "asked_at": time_el.get("datetime") if time_el else None,
        "question_score": int(vote_el.get("data-value")) if vote_el and vote_el.get("data-value") else None,
        "question_text": text_or_none(question_body),
        "accepted_answer_text": text_or_none(accepted_body),
        "code_blocks": code_blocks,
    }

That parser gives you exactly what most internal knowledge workflows need: a structured question, the accepted answer, and the embedded code examples.

Step 4: Put the crawl together

def crawl_tag(tag: str, pages: int = 2) -> list[dict]:
    client = HttpClient()
    all_records = []
    seen_urls = set()

    for page in range(1, pages + 1):
        listing_url = f"{BASE}/questions/tagged/{tag}?tab=Newest&page={page}&pagesize=15"
        listing_html = client.get_html(listing_url)
        summaries = parse_tag_page(listing_html)

        for summary in summaries:
            if summary["url"] in seen_urls:
                continue
            seen_urls.add(summary["url"])

            detail_html = client.get_html(summary["url"])
            detail = parse_question_page(detail_html)
            all_records.append({**summary, **detail})
            time.sleep(0.8)

        print(f"page={page} records={len(all_records)}")

    return all_records

Exporting to JSON is straightforward:

import json

records = crawl_tag("python", pages=1)
with open("stackoverflow_python.json", "w", encoding="utf-8") as fh:
    json.dump(records, fh, ensure_ascii=False, indent=2)

Why this approach is resilient

It uses semantic selectors instead of visual class chains.
It handles tag pages and detail pages separately.
It explicitly looks for accepted-answer indicators instead of assuming the first answer is best.
It collects code blocks in the same pass, which saves a second parse later.

If Stack Overflow changes a CSS class, your first response should not be "switch to Selenium." It should be "re-open the HTML and fix the narrow selector that drifted."

Practical notes

Respect rate limits. Public pages are readable, but hammering many question pages quickly is still a bad idea.
If you only need metadata, use the Stack Exchange API. Scraping is useful when you need rendered HTML, accepted-answer formatting, or code blocks exactly as shown on page.
Keep question URLs as your natural primary key.
Validate output with a few hand-checked pages before running broad crawls.

For support-search datasets, this pattern is often enough: crawl a curated list of tags nightly, store accepted answers, and refresh only new or recently edited questions.

Scale Stack Overflow crawls without rebuilding your network layer

Stack Overflow is easy to scrape in small batches. When you fan out across tags and thousands of question pages, ProxiesAPI helps keep retries, IP reputation, and fetch consistency under control.

Get 1,000 free API calls View pricing

Extract Stack Overflow question lists and accepted answers for a tag with robust retries, respectful rate limits, and a validation screenshot. Export to JSON/CSV.

tutorial#python#stack-overflow#web-scraping

Scrape Stack Overflow User Profiles and Badges with Python

Extract reputation, badge counts, top tags, and profile metadata from public Stack Overflow user pages into JSON/CSV with robust selectors and a ProxiesAPI-ready fetch layer.

tutorial#python#stack-overflow#web-scraping

Scrape Stack Overflow Questions and Answers by Tag (Python + ProxiesAPI)

Paginate tag feeds, fetch question pages, and parse title/votes/accepted answer into a clean dataset — with a screenshot proof and production-grade Python.

tutorial#python#stack-overflow#web-scraping

Scrape Stack Overflow Questions and Answers by Tag (Python + ProxiesAPI)

Collect Stack Overflow Q&A for a tag with pagination, answer extraction, and a proof screenshot. Export clean JSON for analysis.

tutorial#python#stack-overflow#web-scraping

Scrape Stack Overflow Questions and Answers

Related guides