How to Scrape Shopify Stores: Product, Price, Inventory
If you are searching for how to scrape Shopify stores, the good news is that many stores expose a clean starting point already.
The bad news is that people often overcomplicate the job immediately.
For most catalog-monitoring workflows, you do not need a browser first. You need a practical plan for:
- product titles
- prices
- variants
- availability signals
- incremental refresh
This guide shows the simplest path that works.
Shopify scraping is straightforward on one store and operationally messy on many. ProxiesAPI helps you separate fetch stability from parsing logic as your monitoring footprint grows.
The three Shopify data surfaces that matter
When people ask how to scrape Shopify stores, they usually mean one of three things:
| Surface | Best for | Tradeoff |
|---|---|---|
/products.json | catalog, variants, prices, availability | not every store exposes the full catalog forever |
| product HTML + JSON-LD | validating visible product fields | one request per product page |
| rendered DOM / browser flow | JS-only widgets, cart behavior, region switching | heavier and slower |
The right default is almost always /products.json.
For many stores, a request like this works:
https://example-store.com/products.json?limit=250
That response typically includes:
- product title
- vendor
- handle
- variants
- prices
availablebooleans
That is already enough to build price tracking and basic stock monitoring.
Step 1: Start with products.json
Here is a minimal Python fetcher:
from __future__ import annotations
import os
import random
import time
from urllib.parse import quote_plus
import requests
TIMEOUT = (10, 30)
PROXIESAPI_KEY = os.getenv("PROXIESAPI_KEY", "")
session = requests.Session()
session.headers.update({
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
})
def proxiesapi_url(target_url: str) -> str:
if not PROXIESAPI_KEY:
raise RuntimeError("Set PROXIESAPI_KEY before enabling ProxiesAPI")
return (
"https://api.proxiesapi.com/?auth_key="
+ quote_plus(PROXIESAPI_KEY)
+ "&url="
+ quote_plus(target_url)
)
def fetch_json(url: str, *, use_proxiesapi: bool = False, attempts: int = 4) -> dict:
target = proxiesapi_url(url) if use_proxiesapi else url
last_error = None
for attempt in range(1, attempts + 1):
try:
response = session.get(target, timeout=TIMEOUT)
response.raise_for_status()
return response.json()
except Exception as exc:
last_error = exc
time.sleep(min(10, 2 ** (attempt - 1)) + random.random())
raise RuntimeError(f"fetch failed: {last_error}")
Now build the endpoint:
def products_json_url(store_base: str, *, page: int = 1, limit: int = 250) -> str:
store_base = store_base.rstrip("/")
return f"{store_base}/products.json?limit={limit}&page={page}"
Step 2: Flatten products and variants
Shopify responses are nested. The useful dataset is usually variant-level, not product-level.
def normalize_variant(product: dict, variant: dict) -> dict:
return {
"product_id": product.get("id"),
"product_title": product.get("title"),
"handle": product.get("handle"),
"vendor": product.get("vendor"),
"product_type": product.get("product_type"),
"variant_id": variant.get("id"),
"variant_title": variant.get("title"),
"sku": variant.get("sku"),
"price": variant.get("price"),
"compare_at_price": variant.get("compare_at_price"),
"available": variant.get("available"),
"requires_shipping": variant.get("requires_shipping"),
"created_at": product.get("created_at"),
"updated_at": product.get("updated_at"),
"product_url": f"/products/{product.get('handle')}",
}
def parse_products_payload(payload: dict) -> list[dict]:
rows = []
for product in payload.get("products", []):
variants = product.get("variants", [])
for variant in variants:
rows.append(normalize_variant(product, variant))
return rows
That is enough to build a useful tracker immediately.
Step 3: Paginate until the catalog ends
Some stores fit in one page. Many do not.
def scrape_store_catalog(store_base: str, *, use_proxiesapi: bool = False) -> list[dict]:
rows: list[dict] = []
seen_variant_ids = set()
page = 1
while True:
payload = fetch_json(products_json_url(store_base, page=page), use_proxiesapi=use_proxiesapi)
batch = parse_products_payload(payload)
if not batch:
break
for row in batch:
variant_id = row.get("variant_id")
if variant_id in seen_variant_ids:
continue
seen_variant_ids.add(variant_id)
rows.append(row)
page += 1
time.sleep(random.uniform(0.4, 1.0))
return rows
The key is not the loop. The key is deduping by variant_id, because variant-level tracking is what makes inventory and price monitoring useful.
Step 4: Treat inventory as a signal, not a perfect truth
Many Shopify scrapers overpromise here.
What you can usually get reliably:
variant["available"]- whether a product page says “sold out”
- whether a size/color option disappears
What you often cannot get reliably from public scraping alone:
- exact warehouse counts
- reserved inventory
- future restock allocations
So the right mental model is:
| Signal | Reliability | Use case |
|---|---|---|
available = true/false | high | in-stock / out-of-stock monitoring |
| price change | high | competitor tracking |
| compare-at price | medium-high | discount tracking |
| exact quantity | low | avoid promising this publicly |
If you promise exact stock counts from public Shopify pages, you will disappoint yourself later.
Step 5: Add a product-page fallback
Sometimes a store limits the JSON surface or you want to validate what is visible on the product page. In that case, parse the product HTML and its structured data.
from bs4 import BeautifulSoup
import json
def fetch_html(url: str, *, use_proxiesapi: bool = False) -> str:
target = proxiesapi_url(url) if use_proxiesapi else url
response = session.get(target, timeout=TIMEOUT)
response.raise_for_status()
return response.text
def parse_product_json_ld(html: str) -> list[dict]:
soup = BeautifulSoup(html, "lxml")
out = []
for script in soup.select('script[type="application/ld+json"]'):
raw = script.get_text(strip=True)
if not raw:
continue
try:
payload = json.loads(raw)
except Exception:
continue
objects = payload if isinstance(payload, list) else [payload]
for obj in objects:
if obj.get("@type") != "Product":
continue
offers = obj.get("offers", [])
if isinstance(offers, dict):
offers = [offers]
for offer in offers:
out.append({
"title": obj.get("name"),
"price": offer.get("price"),
"currency": offer.get("priceCurrency"),
"availability": offer.get("availability"),
})
return out
This is not your primary path. It is your validation path and fallback.
Practical workflow for real stores
If your goal is a working Shopify tracker, use this order:
- try
/products.json - flatten variants
- store the
updated_atfield - revisit only changed products
- use product-page HTML only when the JSON surface is incomplete
That sequence is faster, cheaper, and easier to maintain than starting with browser automation.
Export a clean dataset
import pandas as pd
def main() -> None:
store = "https://www.allbirds.com"
rows = scrape_store_catalog(store, use_proxiesapi=bool(PROXIESAPI_KEY))
df = pd.DataFrame(rows)
df["store"] = store
df.to_csv("shopify-catalog.csv", index=False)
print(df[["product_title", "variant_title", "price", "available"]].head(20))
if __name__ == "__main__":
main()
That single CSV can feed:
- price alerts
- out-of-stock alerts
- promo tracking
- competitive assortment analysis
When ProxiesAPI matters
If you monitor one store, maybe never.
If you monitor many stores across many categories, it starts to matter because you need:
- stable request quality
- retries without reworking the parser
- separation between crawl logic and routing logic
That is the real theme here: Shopify parsing is the easy part. Operating many repetitive fetches is the part that becomes annoying.
Final thoughts
The best answer to how to scrape Shopify stores is not “launch a browser and click around.”
It is:
- start with
/products.json - flatten variants
- treat inventory as a public signal, not a guaranteed count
- use product-page JSON-LD as a fallback
That gets you a practical product, price, and availability dataset quickly, and it leaves room to add heavier tooling only when a store actually forces you to.
Shopify scraping is straightforward on one store and operationally messy on many. ProxiesAPI helps you separate fetch stability from parsing logic as your monitoring footprint grows.