Infinite Scroll Scraping: 4 Reliable Patterns for Modern Sites

Infinite scroll scraping is often described as a browser problem.

Usually it is a pagination problem wearing a JavaScript costume.

The page looks like endless scrolling, but underneath it almost always uses one of a few repeatable data-loading patterns. If you identify the pattern first, your scraper gets simpler, faster, and much easier to maintain.

This guide covers four reliable patterns for infinite scroll scraping and when to use each.

Pattern 1: Hidden JSON API behind the scroll

This is the best-case scenario.

The frontend scrolls, then requests JSON from an endpoint such as:

• /api/search?page=2
• /graphql
• /feed?cursor=abc123
• /items?offset=48&limit=24

If you can reproduce that request directly, you do not need to scroll at all.

How to spot it

• Open DevTools Network
• Filter for fetch or xhr
• Scroll once
• Look for a new request that returns structured JSON

Python example

import requests

session = requests.Session()
session.headers.update(
    {
        "User-Agent": (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/126.0.0.0 Safari/537.36"
        )
    }
)

cursor = None
rows = []

while True:
    params = {"limit": 24}
    if cursor:
        params["cursor"] = cursor

    resp = session.get("https://example.com/api/listings", params=params, timeout=(10, 30))
    resp.raise_for_status()
    payload = resp.json()

    rows.extend(payload["items"])
    cursor = payload.get("next_cursor")
    if not cursor:
        break

This is the fastest and cheapest pattern by far.

Pattern 2: Cursor-based HTML fragments

Some sites do not return full JSON. They return server-rendered HTML fragments for "load more" actions.

Typical signs:

• a request includes cursor, page, offset, or last_id
• the response contains partial HTML cards instead of JSON
• the main page injects the new cards into the DOM

Why it is still good

You can often request those fragments directly and parse them with BeautifulSoup, which is still much lighter than running a browser loop.

from bs4 import BeautifulSoup
import requests

session = requests.Session()
cursor = None
cards = []

while True:
    params = {"cursor": cursor} if cursor else {}
    resp = session.get("https://example.com/search/fragment", params=params, timeout=(10, 30))
    resp.raise_for_status()

    soup = BeautifulSoup(resp.text, "lxml")
    batch = soup.select("article.result-card")
    if not batch:
        break

    for card in batch:
        cards.append(
            {
                "title": card.select_one("h2").get_text(" ", strip=True),
                "url": card.select_one("a")["href"],
            }
        )

    next_input = soup.select_one('input[name="cursor"]')
    cursor = next_input.get("value") if next_input else None
    if not cursor:
        break

If the data arrives as HTML, scrape the fragment directly instead of simulating the scroll.

Pattern 3: GraphQL connections with hasNextPage

A lot of modern infinite scroll UIs are backed by GraphQL.

The UI scrolls, then sends:

• a query name
• variables
• a cursor

And the response includes:

• edges
• pageInfo.endCursor
• pageInfo.hasNextPage

Example

import requests

session = requests.Session()
graphql_url = "https://example.com/graphql"

query = """
query SearchFeed($after: String) {
  feed(after: $after, first: 20) {
    edges {
      node {
        id
        title
        url
      }
    }
    pageInfo {
      endCursor
      hasNextPage
    }
  }
}
"""

after = None
items = []

while True:
    payload = {"query": query, "variables": {"after": after}}
    resp = session.post(graphql_url, json=payload, timeout=(10, 30))
    resp.raise_for_status()
    data = resp.json()["data"]["feed"]

    for edge in data["edges"]:
        items.append(edge["node"])

    if not data["pageInfo"]["hasNextPage"]:
        break
    after = data["pageInfo"]["endCursor"]

This is still pagination. It just happens to be GraphQL-shaped.

Pattern 4: Real browser scrolling as a fallback

Sometimes there is no easy direct endpoint, or the site aggressively ties content loading to browser execution. Then a browser is justified.

Use Playwright if:

• content only loads after actual scroll events
• the request signatures are hard to reproduce
• a bot challenge or dynamic app state breaks direct replay
• you need rendered content, not just transport data

Playwright scrolling loop

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/feed", wait_until="networkidle")

    seen = set()

    for _ in range(30):
        cards = page.locator("article.result-card")
        count = cards.count()

        for i in range(count):
            href = cards.nth(i).locator("a").first.get_attribute("href")
            if href:
                seen.add(href)

        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        page.wait_for_timeout(1500)

    print(f"collected {len(seen)} unique items")
    browser.close()

This works, but it should be your last resort, not the first instinct.

Comparison table: which pattern should you prefer?

The general rule is simple: scrape the transport layer before you scrape the viewport.

Debug checklist for infinite scroll scraping

When a scroll scraper feels flaky, ask these questions:

1. Is the site loading JSON, HTML fragments, or GraphQL?
2. What parameter moves pagination forward: page, offset, cursor, token, or timestamp?
3. Does the response itself reveal the next cursor?
4. Is duplicate data appearing because the cursor logic is wrong?
5. Are you scrolling because you have to, or because you have not inspected the network yet?

That checklist solves most "infinite scroll scraping" problems before code changes even begin.

Where ProxiesAPI fits

Infinite scroll workflows often multiply request volume fast:

• one page becomes dozens of follow-up calls
• cursors increase total runtime
• retries compound under throttling

ProxiesAPI helps when you need a cleaner network layer around that request volume. It will not reveal the site's pagination pattern for you, but once you understand the pattern, it can make collection more stable.

The practical recommendation

For modern sites, start with Network tab archaeology, not browser automation. Find the real pagination shape. Reproduce it directly if you can. Only fall back to a scrolling browser when transport-level scraping is genuinely blocked.

That is how you turn infinite scroll scraping from a flaky demo into something you can run in production.