Scrape Shopee Seller Storefronts and Top Products with Python
Shopee storefront scraping in 2026 is not the old "requests plus BeautifulSoup" problem. The shop pages are heavily JavaScript-driven, some regions gate anonymous traffic, and the same URL can behave differently depending on locale, cookies, and device context.
That doesn't mean the job is impossible. It means the reliable workflow is:
- open the storefront in a browser
- capture the shop identity from the page bootstrap
- collect top-product cards from the rendered DOM when they load
- keep your parser honest about failures and partial loads
This guide shows exactly that pattern.
Shopee storefront scraping is inconsistent across regions and anonymous sessions. ProxiesAPI gives you a cleaner way to route requests once you've identified the storefront and data endpoints you need.
The reality of Shopee storefronts
A live Shopee storefront page still exposes useful signals even when the product grid is JS-rendered:
- the page bootstrap includes shop metadata
- the URL gives you the market and seller path
- rendered product cards can be collected with a browser once the grid loads
- failure states are explicit, which lets you retry or mark the run incomplete
During validation for this post, anonymous storefront sessions returned a visible "This shop failed to load" message in some regions. That is exactly why a browser-assisted flow beats pretending every response is a complete HTML page.
Install dependencies
python -m venv .venv
source .venv/bin/activate
pip install playwright beautifulsoup4 requests
playwright install chromium
We'll use:
playwrightfor the rendered storefrontrequestsfor optional follow-up callsBeautifulSoupfor parsing the page source when needed
Step 1: Fetch the storefront in a real browser
We'll start with a Playwright session so Shopee can execute its JS and populate bootstrap variables.
import json
from dataclasses import dataclass, asdict
from playwright.sync_api import sync_playwright
@dataclass
class ProductCard:
title: str | None
price: str | None
href: str | None
def open_storefront(url: str) -> dict:
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page(
viewport={"width": 1440, "height": 2200},
user_agent=(
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
),
)
page.goto(url, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(6000)
bootstrap = page.evaluate(
"""() => {
const assets = window.__ASSETS__ || {};
return assets.MART_CONFIG?.shop || null;
}"""
)
html = page.content()
failed = page.locator("text=This shop failed to load").count() > 0
cards = []
for card in page.locator("a[href*='/product/']").all()[:12]:
cards.append(
ProductCard(
title=card.get_attribute("title") or card.inner_text().strip()[:160],
price=None,
href=card.get_attribute("href"),
)
)
browser.close()
return {
"bootstrap_shop": bootstrap,
"failed_to_load": failed,
"cards": [asdict(card) for card in cards],
"html": html,
}
Why this works:
window.__ASSETS__is present in live Shopee responsesMART_CONFIG.shopoften exposes ashopidandusername- DOM collection gives you the product grid when the storefront actually renders
Even if the grid fails, the scraper still tells you why instead of silently returning an empty list.
Step 2: Parse bootstrap metadata and storefront status
Here's a small wrapper that converts the browser result into a cleaner record:
from urllib.parse import urljoin
def normalize_storefront(url: str) -> dict:
result = open_storefront(url)
shop = result.get("bootstrap_shop") or {}
return {
"storefront_url": url,
"shopid": shop.get("shopid"),
"username": shop.get("username"),
"failed_to_load": result["failed_to_load"],
"product_count_collected": len(result["cards"]),
"top_products": [
{
"title": card["title"],
"url": urljoin(url, card["href"]) if card["href"] else None,
}
for card in result["cards"]
],
}
If a shop partially loads, you still get:
- the seller identity from bootstrap
- whether the page was a failure state
- however many product links rendered before the timeout
That is much more useful in production than a binary success/fail script.
Step 3: Add optional ProxiesAPI-backed follow-up requests
Once you've discovered the storefront and collected product URLs, you may want to fetch individual product pages or related public pages through a stable request layer.
import os
from urllib.parse import quote
import requests
PROXIESAPI_KEY = os.getenv("PROXIESAPI_KEY")
session = requests.Session()
def proxiesapi_url(target_url: str) -> str:
if not PROXIESAPI_KEY:
return target_url
return f"https://api.proxiesapi.com/?auth_key={quote(PROXIESAPI_KEY)}&url={quote(target_url, safe='')}"
def fetch_followup(url: str) -> str:
response = session.get(
proxiesapi_url(url),
headers={
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
)
},
timeout=(15, 45),
)
response.raise_for_status()
return response.text
The honest use case here is not "ProxiesAPI magically unlocks every Shopee endpoint." The real value is that it gives your pipeline one predictable outbound layer when you start doing lots of follow-up requests across products and markets.
Step 4: Save seller records as JSON
def scrape_storefronts(urls: list[str], output_path: str = "shopee_storefronts.json") -> None:
records = []
for url in urls:
record = normalize_storefront(url)
print(
f"{url} -> shopid={record['shopid']} failed={record['failed_to_load']} "
f"products={record['product_count_collected']}"
)
records.append(record)
with open(output_path, "w", encoding="utf-8") as f:
json.dump(records, f, indent=2, ensure_ascii=False)
if __name__ == "__main__":
scrape_storefronts(
[
"https://shopee.sg/smartsg",
"https://shopee.ph/shop/251015284",
]
)
Example output shape:
{
"storefront_url": "https://shopee.sg/smartsg",
"shopid": 91799978,
"username": "smartsg",
"failed_to_load": true,
"product_count_collected": 0,
"top_products": []
}
That sample is not a bug. It's the exact kind of operational truth you want from a storefront scraper: whether the page genuinely loaded and whether product cards were available in that session.
What to do when the storefront fails
Shopee storefronts can fail for reasons that have nothing to do with your parser:
- market-level anonymous restrictions
- anti-bot friction
- region mismatch between your IP and the target marketplace
- storefront components that only hydrate after additional API calls
Three practical fixes:
- retry with a real browser and a longer wait budget
- test a market-local exit point if you're scraping region-specific stores
- split the job into storefront discovery and product-page scraping instead of forcing everything through one page
If your business goal is catalog intelligence, the third option is often the cleanest.
Why this pattern beats raw HTML scraping
Shopee is a good example of modern scraping reality:
- the bootstrap tells you who the seller is
- the browser tells you whether the storefront actually rendered
- your output should preserve failure states instead of hiding them
That makes the resulting dataset trustworthy. And once you have trustworthy seller discovery, you can use the same pipeline to branch into product pages, price tracking, and availability checks with ProxiesAPI handling the repetitive network layer.
Shopee storefront scraping is inconsistent across regions and anonymous sessions. ProxiesAPI gives you a cleaner way to route requests once you've identified the storefront and data endpoints you need.