Steam Scraper: Extract Prices, Reviews, and Tags with Python
If you are building a Steam scraper, the good news is that a lot of useful metadata is still visible in server-rendered HTML.
You can reliably collect:
- title
- app id
- release date
- search-result price
- review count
- review summary
- user-facing tags
The trick is not to expect every field from one page. The listing page and the app page each tell a different part of the story.
This guide shows a practical Steam scraper in Python that starts from search results, follows app links, and exports a dataset you can actually analyze.
Steam pages are usable with plain HTTP requests, but once you broaden the crawl across many searches and app pages, ProxiesAPI can stabilize the fetch layer while your parser stays unchanged.
The two-page model
For a Steam scraper, think in two layers:
| Page type | Best fields | Useful selectors |
|---|---|---|
| search results | title, app id, release date, listing price | a.search_result_row, span.title, .search_released, .discount_final_price |
| app page | tags, description, review summary, review count | a.app_tag, div.game_description_snippet, span.game_review_summary, meta[itemprop="reviewCount"] |
That split is important because it keeps your crawler honest. The list page is for discovery. The app page is for enrichment.
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml pandas
Step 1: Fetch Steam pages with optional ProxiesAPI
from __future__ import annotations
import os
import random
import time
from typing import Optional
import requests
TIMEOUT = (10, 40)
PROXIESAPI_PROXY_URL = os.getenv("PROXIESAPI_PROXY_URL")
session = requests.Session()
session.headers.update(
{
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
)
def proxy_config() -> Optional[dict[str, str]]:
if not PROXIESAPI_PROXY_URL:
return None
return {"http": PROXIESAPI_PROXY_URL, "https": PROXIESAPI_PROXY_URL}
def fetch(url: str, *, max_retries: int = 4) -> str:
last_error = None
for attempt in range(1, max_retries + 1):
try:
response = session.get(url, timeout=TIMEOUT, proxies=proxy_config())
if response.status_code in (403, 429, 500, 502, 503, 504):
raise requests.HTTPError(f"retryable status {response.status_code}")
response.raise_for_status()
return response.text
except Exception as exc: # noqa: BLE001
last_error = exc
time.sleep(min(12, attempt * 2) + random.random())
raise RuntimeError(f"failed to fetch {url}: {last_error}")
The code works without ProxiesAPI. The env var only becomes useful once the crawl stops being tiny.
Step 2: Parse search results
A search page gives you the discovery layer.
Example URL:
https://store.steampowered.com/search/?term=portal&supportedlang=english
import re
from bs4 import BeautifulSoup
def clean(text: str | None) -> str | None:
if not text:
return None
value = re.sub(r"\s+", " ", text).strip()
return value or None
def parse_search_results(html: str) -> list[dict]:
soup = BeautifulSoup(html, "lxml")
rows: list[dict] = []
for row in soup.select("a.search_result_row"):
title_el = row.select_one("span.title")
title = clean(title_el.get_text(" ", strip=True) if title_el else None)
if not title:
continue
price_el = row.select_one(".discount_final_price")
release_el = row.select_one(".search_released")
rows.append(
{
"appid": row.get("data-ds-appid"),
"title": title,
"store_url": row.get("href", "").split("?")[0],
"release_date": clean(release_el.get_text(" ", strip=True) if release_el else None),
"listing_price": clean(price_el.get_text(" ", strip=True) if price_el else None),
}
)
return rows
That alone is enough for a basic market scan. But it does not yet give you reviews or tags.
Step 3: Enrich each app page
This is where the Steam scraper gets more useful.
def parse_app_page(html: str) -> dict:
soup = BeautifulSoup(html, "lxml")
description_el = soup.select_one("div.game_description_snippet")
review_summary_el = soup.select_one("span.game_review_summary")
review_count_el = soup.select_one('meta[itemprop="reviewCount"]')
tags = []
for tag in soup.select("a.app_tag"):
value = clean(tag.get_text(" ", strip=True))
if value and value not in tags:
tags.append(value)
if len(tags) >= 8:
break
return {
"description": clean(description_el.get_text(" ", strip=True) if description_el else None),
"review_summary": clean(review_summary_el.get_text(" ", strip=True) if review_summary_el else None),
"review_count": review_count_el.get("content") if review_count_el else None,
"tags": tags,
}
def enrich_rows(rows: list[dict]) -> list[dict]:
enriched = []
for row in rows:
html = fetch(row["store_url"])
details = parse_app_page(html)
enriched.append({**row, **details})
time.sleep(random.uniform(0.8, 1.8))
return enriched
The selector choices are grounded in the current Steam HTML:
a.app_tagdiv.game_description_snippetspan.game_review_summarymeta[itemprop="reviewCount"]
That last selector is especially useful because a meta tag is often less noisy than scraping review count text from a larger block.
Step 4: Put the Steam scraper together
import pandas as pd
from urllib.parse import quote_plus
def search_url(term: str) -> str:
return f"https://store.steampowered.com/search/?term={quote_plus(term)}&supportedlang=english"
def scrape_steam(term: str, *, limit: int = 10) -> pd.DataFrame:
html = fetch(search_url(term))
rows = parse_search_results(html)[:limit]
rows = enrich_rows(rows)
df = pd.DataFrame(rows)
if not df.empty:
df["tag_string"] = df["tags"].apply(lambda items: ", ".join(items))
return df
if __name__ == "__main__":
df = scrape_steam("portal", limit=5)
print(df[["title", "listing_price", "review_summary", "review_count", "tag_string"]].to_string(index=False))
df.to_csv("steam_games.csv", index=False)
print("saved steam_games.csv rows:", len(df))
Example output shape:
title listing_price review_summary review_count tag_string
Portal ₹480.00 Overwhelmingly Positive 83044 Puzzle, Puzzle Platformer, First-Person
Portal 2 Sixense MotionPack None Very Positive 1068 Action, Adventure, Puzzle
Practical advice
| Situation | Better move |
|---|---|
| you need broad discovery | scrape search results first |
| you need richer metadata | visit the app page second |
| you need hundreds of apps | batch runs and cache app HTML |
| you need region-consistent prices | keep locale and network settings stable across runs |
Two more notes:
- Steam prices can differ by currency and region, so do not compare runs taken from different locales as if they were identical.
- Some games are free, discounted, delisted, or packaged in bundles. Handle missing price fields without crashing the scraper.
Where ProxiesAPI fits
For a small run of five or ten apps, you may not need anything beyond normal requests.
ProxiesAPI becomes more relevant when you start:
- running many search terms daily
- revisiting app pages repeatedly
- collecting multiple locales or categories
Keep it in the fetch layer so your Steam scraper stays easy to reason about.
Wrap-up
A practical Steam scraper is not one giant parser. It is a small two-step pipeline:
- discover games from search results
- enrich games from app pages
That gives you prices, reviews, and tags without making the crawler brittle from day one.
Steam pages are usable with plain HTTP requests, but once you broaden the crawl across many searches and app pages, ProxiesAPI can stabilize the fetch layer while your parser stays unchanged.