Scraping Hidden APIs: How to Find the Real Data Behind Modern Sites

A lot of "hard" scraping targets are not actually hard.

They just look hard because you are staring at the rendered page instead of the data request that built it.

Modern frontends often do this:

1. load a small HTML shell
2. call one or more JSON endpoints
3. render cards, tables, or charts from that data

If you can find the request behind the UI, you can often skip brittle DOM scraping entirely.

That is what people usually mean by a hidden API: not a public documented API, but a network endpoint the site itself already uses.

Why hidden APIs matter

Compared with scraping rendered HTML, replaying the real data request is often:

• faster
• cheaper
• easier to paginate
• easier to validate
• less likely to break on cosmetic redesigns

Search results for this topic are remarkably consistent. The dominant workflow is:

• open DevTools
• filter Network to fetch or xhr
• trigger the UI action
• inspect the request
• replay the useful endpoint directly

That is the practical playbook.

The five-step discovery workflow

1. Open the page normally

Load the target page in a browser and use the site the way a real user would.

Do not start by reading minified JavaScript. That is usually the slowest path.

2. Open DevTools Network

Use the browser Network tab and filter to:

• fetch
• xhr

If the site is heavy, also clear the log and repeat the user action once.

3. Trigger the interesting action

Examples:

• click "next page"
• apply a price filter
• open a search result
• scroll once

Watch for new requests that happen exactly when the data on screen changes.

4. Inspect the response body

You are looking for:

• JSON arrays
• GraphQL payloads
• HTML fragments
• cursor or page parameters

The winning request is usually the one that returns the actual records, not the one that loads fonts, ads, analytics, or images.

5. Replay it outside the browser

Once you have the URL, query params, headers, cookies, and body shape, reproduce it in code.

If you can do that, the browser becomes a discovery tool instead of a runtime dependency.

Pattern 1: Straightforward JSON endpoint

Best case:

• request: /api/search?q=headphones&page=2
• response: JSON with items

import requests

session = requests.Session()
session.headers.update(
    {
        "User-Agent": (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/126.0.0.0 Safari/537.36"
        ),
        "Accept": "application/json,text/plain,*/*",
    }
)

resp = session.get(
    "https://example.com/api/search",
    params={"q": "headphones", "page": 2},
    timeout=(10, 30),
)
resp.raise_for_status()
payload = resp.json()
print(len(payload["items"]))

This is the fastest possible outcome because the page has already done the reverse engineering for you.

Pattern 2: Cursor-based API

Many modern sites use cursors instead of page numbers.

You will see parameters like:

• cursor
• next
• after
• last_id

def collect_all_items() -> list[dict]:
    cursor = None
    rows = []

    while True:
        params = {"limit": 24}
        if cursor:
            params["cursor"] = cursor

        resp = session.get("https://example.com/api/listings", params=params, timeout=(10, 30))
        resp.raise_for_status()
        payload = resp.json()

        rows.extend(payload["items"])
        cursor = payload.get("next_cursor")
        if not cursor:
            break

    return rows

This is much easier to maintain than simulating scroll events forever.

Pattern 3: GraphQL request

Some sites use a single endpoint such as /graphql with a JSON body.

That is still scrape-friendly once you capture:

• operation name
• query or persisted query id
• variables

graphql_payload = {
    "operationName": "SearchProducts",
    "variables": {"query": "headphones", "page": 1},
    "query": """
        query SearchProducts($query: String!, $page: Int!) {
          search(query: $query, page: $page) {
            items { id name price }
            pageInfo { hasNextPage }
          }
        }
    """,
}

resp = session.post("https://example.com/graphql", json=graphql_payload, timeout=(10, 30))
resp.raise_for_status()
data = resp.json()

GraphQL looks intimidating the first time, but operationally it is just another HTTP request with a JSON response.

Pattern 4: HTML fragments instead of JSON

Not every hidden data endpoint returns JSON. Some return partial HTML that the frontend injects into the page.

That is still better than full browser automation.

from bs4 import BeautifulSoup

resp = session.get("https://example.com/search/fragment", params={"page": 2}, timeout=(10, 30))
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "lxml")
cards = []
for card in soup.select("article.product-card"):
    cards.append(
        {
            "title": card.select_one("h2").get_text(" ", strip=True),
            "url": card.select_one("a")["href"],
        }
    )

You still skip scrolling, clicking, and rendering overhead.

Comparison: DOM scraping vs hidden API scraping

A good rule is:

Try hidden API replay before you commit to browser scraping.

A real discovery workflow with Playwright

You can use Playwright once for discovery, then switch back to requests for production.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    def log_response(response):
        if response.request.resource_type in {"fetch", "xhr"}:
            print(response.status, response.url)

    page.on("response", log_response)
    page.goto("https://example.com/search?q=headphones", wait_until="networkidle")
    page.click("text=Next")
    page.wait_for_timeout(3000)
    browser.close()

This is enough to reveal the requests you actually care about.

Practical advice for replaying the request

When an endpoint fails outside the browser, check these in order:

1. missing query params
2. missing headers such as x-requested-with or content-type
3. required cookies
4. anti-CSRF token
5. wrong HTTP method
6. stale cursor or page token

Do not blindly copy every browser header. Start with the minimum set that makes the request work.

Where ProxiesAPI helps

Once you find the data endpoint, ProxiesAPI is often more useful there than on the rendered page:

• fewer bytes transferred
• fewer moving parts
• easier retries
• easier pagination loops

That means the cheapest production architecture is often:

1. browser for discovery
2. direct HTTP for production
3. ProxiesAPI when the request volume or network quality needs help

When hidden APIs are not enough

Use a full browser at runtime when:

• data only appears after complex user interaction
• the endpoint requires short-lived client-generated tokens
• the site signs requests in JavaScript
• content is truly rendered from browser-only state

Even then, the hidden API workflow still helps because it tells you what the page is trying to do.

Final thoughts

Hidden API scraping is not a trick. It is just paying attention to how modern sites already move data.

Before you write a fragile DOM scraper, ask:

• what request created these cards?
• what params control pagination?
• is the response already clean JSON?

When the answer is yes, scrape the data pipe instead of the paint.