Scrape Live Stock Data from Yahoo Finance
Yahoo Finance is one of the fastest ways to turn a personal watchlist into a dataset you control.
In this guide we will build a Python scraper that:
- fetches real Yahoo Finance quote pages
- extracts the visible quote snapshot for each ticker
- pulls a few summary rows from the same page
- exports clean CSV for downstream alerts or dashboards
Mandatory screenshot of the target site:
Quote pages are fine at small volume, but watchlists quickly become repetitive, bursty traffic. ProxiesAPI gives you a cleaner fetch layer when you scale beyond manual checks.
What we are scraping
We will use standard Yahoo Finance quote URLs such as:
https://finance.yahoo.com/quote/AAPLhttps://finance.yahoo.com/quote/MSFThttps://finance.yahoo.com/quote/NVDA
The page gives us two useful surfaces:
- visible quote widgets rendered as
fin-streamerelements - a large embedded JSON payload that includes the same quote data plus summary fields
That combination is practical. We can keep the parser simple, validate against what the user sees, and avoid reverse-engineering a private API flow for a basic watchlist job.
This is still scraping, not a licensed market-data feed. Treat it as a learning or internal-monitoring workflow, not a compliance-grade trading input.
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml pandas
We will use:
requestsfor HTTPBeautifulSoupfor light HTML extractionpandasfor CSV export
Step 1: Build a ProxiesAPI-ready fetch layer
Even a small quote monitor benefits from explicit timeouts, retries, and a clean user agent.
from __future__ import annotations
import os
import random
import time
from urllib.parse import quote_plus
import requests
TIMEOUT = (10, 30)
PROXIESAPI_KEY = os.getenv("PROXIESAPI_KEY", "")
session = requests.Session()
session.headers.update({
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
})
def proxiesapi_url(target_url: str) -> str:
if not PROXIESAPI_KEY:
raise RuntimeError("Set PROXIESAPI_KEY before enabling ProxiesAPI")
return (
"https://api.proxiesapi.com/?auth_key="
+ quote_plus(PROXIESAPI_KEY)
+ "&url="
+ quote_plus(target_url)
)
def fetch_html(url: str, *, use_proxiesapi: bool = False, attempts: int = 4) -> str:
last_error = None
for attempt in range(1, attempts + 1):
try:
target = proxiesapi_url(url) if use_proxiesapi else url
response = session.get(target, timeout=TIMEOUT)
response.raise_for_status()
html = response.text
if "Will be right back" in html or len(html) < 100_000:
raise RuntimeError(f"unexpected HTML size: {len(html)}")
return html
except Exception as exc:
last_error = exc
time.sleep(min(12, 2 ** (attempt - 1)) + random.random())
raise RuntimeError(f"fetch failed for {url}: {last_error}")
If you only scrape a few symbols once a day, direct requests may be enough. Once you start cycling through larger watchlists or intraday refreshes, keeping the proxy layer outside the parser makes life easier.
Step 2: Extract the quote header
Yahoo renders the visible price with fin-streamer nodes. The page contains other market widgets too, so we keep the selector narrow by first finding the quote title and then reading the first visible quote block.
from bs4 import BeautifulSoup
def parse_quote_header(html: str) -> dict:
soup = BeautifulSoup(html, "lxml")
title = soup.select_one("h1")
name_line = title.get_text(" ", strip=True) if title else None
prices = [el.get_text(" ", strip=True) for el in soup.select('fin-streamer[data-field="regularMarketPrice"]')]
changes = [el.get_text(" ", strip=True) for el in soup.select('fin-streamer[data-field="regularMarketChange"]')]
percents = [el.get_text(" ", strip=True) for el in soup.select('fin-streamer[data-field="regularMarketChangePercent"]')]
return {
"name_line": name_line,
"regular_market_price": prices[0] if prices else None,
"regular_market_change": changes[0] if changes else None,
"regular_market_change_percent": percents[0] if percents else None,
}
On a live page for AAPL, the important visible selectors are:
h1fin-streamer[data-field="regularMarketPrice"]fin-streamer[data-field="regularMarketChange"]fin-streamer[data-field="regularMarketChangePercent"]
These are real page selectors, not guessed class names.
Step 3: Parse summary fields from the embedded page JSON
For fields like volume, market cap, and day range, the embedded page data is more stable than trying to locate a particular table layout.
import json
import re
ROOT_APP_RE = re.compile(r"root\\.App\\.main\\s*=\\s*(\\{.*?\\});\\n", re.DOTALL)
def extract_root_app(html: str) -> dict:
match = ROOT_APP_RE.search(html)
if not match:
raise RuntimeError("Could not find Yahoo root.App.main JSON")
return json.loads(match.group(1))
def parse_summary_payload(payload: dict, symbol: str) -> dict:
stores = payload["context"]["dispatcher"]["stores"]
quote_store = stores["QuoteSummaryStore"]
price = quote_store["price"]
summary = quote_store.get("summaryDetail", {})
return {
"symbol": symbol.upper(),
"currency": price.get("currency"),
"exchange": price.get("exchangeName"),
"market_state": price.get("marketState"),
"market_cap": (summary.get("marketCap") or {}).get("fmt"),
"previous_close": (summary.get("previousClose") or {}).get("fmt"),
"open": (summary.get("open") or {}).get("fmt"),
"day_low": (summary.get("dayLow") or {}).get("fmt"),
"day_high": (summary.get("dayHigh") or {}).get("fmt"),
"volume": (summary.get("volume") or {}).get("fmt"),
"avg_volume": (summary.get("averageVolume") or {}).get("fmt"),
}
Why mix DOM selectors and page JSON?
- the DOM is excellent for the visible headline quote
- the JSON is excellent for secondary stats
That hybrid pattern is common on modern sites.
Step 4: Scrape a whole watchlist
import pandas as pd
def scrape_symbol(symbol: str, *, use_proxiesapi: bool = False) -> dict:
url = f"https://finance.yahoo.com/quote/{symbol.upper()}"
html = fetch_html(url, use_proxiesapi=use_proxiesapi)
header = parse_quote_header(html)
payload = extract_root_app(html)
summary = parse_summary_payload(payload, symbol)
return {
"symbol": symbol.upper(),
"name_line": header["name_line"],
"regular_market_price": header["regular_market_price"],
"regular_market_change": header["regular_market_change"],
"regular_market_change_percent": header["regular_market_change_percent"],
**summary,
"quote_url": url,
}
def scrape_watchlist(symbols: list[str], *, use_proxiesapi: bool = False) -> pd.DataFrame:
rows = []
for symbol in symbols:
rows.append(scrape_symbol(symbol, use_proxiesapi=use_proxiesapi))
time.sleep(random.uniform(0.8, 1.6))
return pd.DataFrame(rows)
Step 5: Export CSV
def main() -> None:
symbols = ["AAPL", "MSFT", "NVDA", "AMZN", "GOOG"]
df = scrape_watchlist(symbols, use_proxiesapi=bool(PROXIESAPI_KEY))
df.to_csv("yahoo-finance-watchlist.csv", index=False)
print(df[["symbol", "regular_market_price", "regular_market_change_percent", "volume"]])
if __name__ == "__main__":
main()
Typical output looks like this:
symbol regular_market_price regular_market_change_percent volume
0 AAPL 213.45 +0.81% 42.1M
1 MSFT 517.08 -0.22% 18.4M
2 NVDA 171.32 +1.94% 66.7M
The exact values change continuously, which is the point of the scraper.
What usually breaks first
Yahoo Finance is manageable, but finance sites are never static forever. The common failure modes are:
- layout changes that move or rename visible widgets
- rate limiting when you hammer many symbols in bursts
- suspiciously small HTML responses that are not real quote pages
The fix is usually operational, not architectural:
- keep timeouts and retries
- cache recent snapshots
- add response-size sanity checks
- use ProxiesAPI when you start rotating through larger symbol sets
When to switch away from page scraping
Use this approach when you need:
- a lightweight watchlist export
- internal monitoring
- a tutorial-friendly dataset
Switch to a dedicated market-data source when you need:
- hard SLA expectations
- deep historical coverage
- legal clarity for redistribution
- sub-second decision making
Scraping quote pages is a solid engineering pattern. It is not a substitute for a licensed trading feed.
Final thoughts
Yahoo Finance is a strong teaching target because it shows two useful scraping habits at once:
- validate against visible page elements
- prefer structured data when the page already exposes it
That gives you a scraper that is easier to debug than pure API reverse-engineering and less fragile than relying on random CSS classes alone.
If you want the smallest reliable starter, begin with five symbols, once-daily snapshots, CSV export, and one sanity alert for short responses. That is enough to build something useful before you scale it up.
Quote pages are fine at small volume, but watchlists quickly become repetitive, bursty traffic. ProxiesAPI gives you a cleaner fetch layer when you scale beyond manual checks.