ProxiesAPIGuides

How to Scrape GitHub Trending with Python (and Export to CSV/JSON)

Mar 10, 2026 · tutorial · #python, #github, #web-scraping, #requests, #beautifulsoup, #csv

GitHub Trending is a simple, high-signal dataset:

  • What repos are spiking today?
  • Which languages are “hot” this week?
  • What should you keep an eye on?

This guide shows how to scrape it without guessing selectors, and export a clean dataset you can use in a cron job.

GitHub Trending page

Turn this into a reliable daily job with ProxiesAPI

Once you run this every day (or across multiple languages/time windows), reliability becomes the bottleneck. ProxiesAPI helps keep the fetch layer stable and predictable.

Get 1,000 free API callsView pricing

Target URL

https://github.com/trending

You can also filter by language and timeframe (same page, query params).

Setup

python -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml

Step 1: Fetch HTML

import requests

URL = "https://github.com/trending"
TIMEOUT = (10, 30)
UA = "Mozilla/5.0 (compatible; ProxiesAPIGuidesBot/1.0; +https://www.proxiesapi.com/)"

html = requests.get(URL, timeout=TIMEOUT, headers={"User-Agent": UA}).text
print(len(html))
print(html[:200])

Step 2: Inspect the DOM (what to extract)

Each repo entry contains:

  • owner/name
  • description (sometimes)
  • language (sometimes)
  • total stars + forks
  • “stars today” (the core metric)

GitHub markup changes over time, so treat selectors as versioned.

The safe approach:

  1. find the list of repo cards
  2. inside each card, extract fields with defensive fallbacks
import re
from bs4 import BeautifulSoup


def to_int(s: str) -> int | None:
    s = (s or "").replace(",", "")
    m = re.search(r"(\d+)", s)
    return int(m.group(1)) if m else None


def scrape_trending(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")

    cards = soup.select('article.Box-row')
    out = []

    for c in cards:
        a = c.select_one('h2 a')
        name = a.get_text(" ", strip=True).replace("\n", " ") if a else None
        href = a.get("href") if a else None
        repo_url = f"https://github.com{href}" if href and href.startswith("/") else href

        desc = None
        p = c.select_one('p')
        if p:
            desc = p.get_text(" ", strip=True)

        lang = None
        lang_el = c.select_one('[itemprop="programmingLanguage"]')
        if lang_el:
            lang = lang_el.get_text(strip=True)

        # total stars/forks
        links = c.select('a.Link--muted')
        stars_total = to_int(links[0].get_text(" ", strip=True)) if len(links) >= 1 else None
        forks_total = to_int(links[1].get_text(" ", strip=True)) if len(links) >= 2 else None

        stars_today = None
        today = c.select_one('span.d-inline-block.float-sm-right')
        if today:
            stars_today = to_int(today.get_text(" ", strip=True))

        out.append({
            "repo": name,
            "url": repo_url,
            "description": desc,
            "language": lang,
            "stars_total": stars_total,
            "forks_total": forks_total,
            "stars_today": stars_today,
        })

    return out


rows = scrape_trending(html)
print("repos:", len(rows))
print(rows[0])

Step 4: Export to CSV + JSON

import csv
import json

with open("github_trending.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
    w.writeheader()
    w.writerows(rows)

with open("github_trending.json", "w", encoding="utf-8") as f:
    json.dump(rows, f, ensure_ascii=False, indent=2)

print("wrote github_trending.csv and github_trending.json")

Running this daily (what actually matters)

  • Dedupe by repo URL
  • Store historical snapshots (date → list) so you can compute deltas
  • Cache the HTML (so debugging doesn’t re-fetch repeatedly)

ProxiesAPI usage (canonical)

When you want to route fetches through ProxiesAPI, use the API-style call:

curl "http://api.proxiesapi.com/?key=API_KEY&url=https://github.com/trending"

(Then your scraper parses the returned HTML.)

QA checklist

  • Parsed ~25 repos
  • Repo URLs are absolute
  • CSV opens cleanly in Sheets
  • Missing fields are handled (language/description)
Turn this into a reliable daily job with ProxiesAPI

Once you run this every day (or across multiple languages/time windows), reliability becomes the bottleneck. ProxiesAPI helps keep the fetch layer stable and predictable.

Get 1,000 free API callsView pricing

Related guides

Scrape GitHub Repository Contributors and Commit Activity with Python
Turn GitHub's public contributors and commit charts into a lightweight engineering-health dataset with Python, CSV export, and practical retry patterns.
tutorial#python#github#web-scraping
Scrape GitHub Repository Stars, Forks, and Topics with Python
Parse public GitHub repository pages into a compact market-intel dataset with stars, forks, topics, and README headings.
tutorial#python#github#web-scraping
Scrape GitHub Topic Pages with Python + ProxiesAPI
Collect repository cards, stars, languages, repo URLs, and update timestamps from GitHub topic pages into a niche-watch dataset.
tutorial#python#github#web-scraping
Scrape GitHub Trending Repositories with Python
Build a daily GitHub Trending dataset with Python: collect repository names, languages, star counts, and URLs, then export clean CSV or JSON with an optional ProxiesAPI fetch layer.
tutorial#python#github#web-scraping