Scrape UK Property Prices from Rightmove
Rightmove is one of the most useful sources for UK housing research because a single crawl can give you listing URLs, asking prices, addresses, agent names, and the raw material for your own market dataset.
The trick is not "can I fetch one page?" It is "can I turn that page into a repeatable pipeline I can run again next week?"
In this guide we will build a Python scraper that:
- starts from a real Rightmove results page
- extracts property detail URLs from listing cards
- parses title, price, address, bedroom count, property type, and agent name
- follows pagination
- exports clean CSV and JSON output
- optionally routes traffic through ProxiesAPI when you need more resilience
Rightmove pages are simple until you scale. ProxiesAPI gives your scraper a stable proxy layer when you need to crawl more pages without juggling IP rotation yourself.
What we are scraping
For a stable tutorial, use Rightmove's property pages as a two-step crawl:
- a search or sold-price results page to discover links
- individual property pages to collect structured fields
Rightmove's markup changes over time, so the safest approach is:
- rely on URL patterns for property links
- parse embedded JSON when it exists
- use HTML selectors as a fallback
Always keep your start URL configurable instead of hard-coding one narrow search into the parser.
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml tenacity pandas
We will use:
requestsfor HTTPBeautifulSoupfor parsingtenacityfor retriespandasonly for the CSV export convenience
Step 1: Build a reliable fetch layer
Property sites often respond with throttling, redirects, or inconsistent HTML if you hammer them. Start with timeouts, headers, and retries.
import os
import random
import time
from dataclasses import dataclass
import requests
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
TIMEOUT = (10, 30)
USER_AGENTS = [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
]
class RetryableFetchError(Exception):
pass
@dataclass
class FetchResult:
url: str
status_code: int
text: str
def proxiesapi_proxy() -> dict[str, str] | None:
key = os.getenv("PROXIESAPI_KEY")
if not key:
return None
proxy = f"http://{key}:@proxy.proxiesapi.com:10000"
return {"http": proxy, "https": proxy}
@retry(
reraise=True,
stop=stop_after_attempt(5),
wait=wait_exponential(multiplier=1, min=1, max=20),
retry=retry_if_exception_type(RetryableFetchError),
)
def fetch_html(session: requests.Session, url: str, use_proxiesapi: bool = False) -> FetchResult:
headers = {
"User-Agent": random.choice(USER_AGENTS),
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-GB,en;q=0.9",
}
proxies = proxiesapi_proxy() if use_proxiesapi else None
response = session.get(url, headers=headers, timeout=TIMEOUT, proxies=proxies)
if response.status_code in (403, 429, 500, 502, 503, 504):
raise RetryableFetchError(f"retryable status {response.status_code} for {url}")
response.raise_for_status()
return FetchResult(url=url, status_code=response.status_code, text=response.text)
def polite_sleep(low: float = 1.0, high: float = 2.5) -> None:
time.sleep(random.uniform(low, high))
If PROXIESAPI_KEY is not set, the scraper still works in direct mode.
Step 2: Extract property links from a results page
Rightmove property pages usually include /properties/<id> in the URL. That is more stable than a fragile CSS class.
import re
from urllib.parse import urljoin
from bs4 import BeautifulSoup
BASE = "https://www.rightmove.co.uk"
PROPERTY_URL_RE = re.compile(r"/properties/\d+")
def extract_property_urls(results_html: str) -> list[str]:
soup = BeautifulSoup(results_html, "lxml")
seen = set()
urls = []
for a in soup.select("a[href]"):
href = a.get("href", "")
match = PROPERTY_URL_RE.search(href)
if not match:
continue
absolute = urljoin(BASE, match.group(0))
if absolute in seen:
continue
seen.add(absolute)
urls.append(absolute)
return urls
That pattern survives markup churn better than selectors tied to card class names.
Step 3: Parse a Rightmove property page
Many listing pages contain structured data in JSON scripts. When that is present, take it first and only fall back to visible HTML.
import json
import re
def first_text(soup: BeautifulSoup, selectors: list[str]) -> str | None:
for selector in selectors:
node = soup.select_one(selector)
if node:
text = " ".join(node.get_text(" ", strip=True).split())
if text:
return text
return None
def extract_json_objects(soup: BeautifulSoup) -> list[dict]:
objects = []
for script in soup.select('script[type="application/ld+json"]'):
raw = script.get_text(strip=True)
if not raw:
continue
try:
loaded = json.loads(raw)
except json.JSONDecodeError:
continue
if isinstance(loaded, dict):
objects.append(loaded)
elif isinstance(loaded, list):
objects.extend(obj for obj in loaded if isinstance(obj, dict))
return objects
def parse_property_page(html: str, url: str) -> dict:
soup = BeautifulSoup(html, "lxml")
json_objects = extract_json_objects(soup)
title = None
address = None
price = None
property_type = None
for obj in json_objects:
if obj.get("@type") in {"Product", "House", "Residence"}:
title = title or obj.get("name")
address = address or obj.get("address", {}).get("streetAddress")
offers = obj.get("offers") or {}
price = price or offers.get("price")
title = title or first_text(soup, ["h1", "[data-testid='title']"])
address = address or first_text(soup, ["h1 + div", "[data-testid='address-label']"])
price = price or first_text(soup, ["[data-testid='price']"])
facts_text = soup.get_text(" ", strip=True)
bedrooms_match = re.search(r"(\d+)\s+bed", facts_text, flags=re.I)
type_match = re.search(r"\b(flat|apartment|terraced|semi-detached|detached|bungalow)\b", facts_text, flags=re.I)
agent_name = first_text(
soup,
["[data-testid='customer-name']",
"[data-testid='branch-name']",
"a[href*='/estate-agents/']"]
)
return {
"url": url,
"title": title,
"address": address,
"price": price,
"bedrooms": int(bedrooms_match.group(1)) if bedrooms_match else None,
"property_type": type_match.group(1).lower() if type_match else property_type,
"agent_name": agent_name,
}
This pattern is intentionally layered:
- structured data first
- visible HTML second
- regex cleanup third
That makes it less brittle than betting everything on one selector.
Step 4: Add pagination
If your start URL returns a normal search result set, you can follow the next-page link until you hit your page cap.
from urllib.parse import urljoin
def extract_next_page(results_html: str, current_url: str) -> str | None:
soup = BeautifulSoup(results_html, "lxml")
node = soup.select_one('a[rel="next"], a[data-test="pagination-next"], a[href*="index="]')
if not node:
return None
href = node.get("href")
return urljoin(current_url, href) if href else None
def crawl_results(start_url: str, pages: int = 3, use_proxiesapi: bool = False) -> list[str]:
session = requests.Session()
current_url = start_url
all_urls: list[str] = []
seen: set[str] = set()
for _ in range(pages):
result = fetch_html(session, current_url, use_proxiesapi=use_proxiesapi)
batch = extract_property_urls(result.text)
for url in batch:
if url in seen:
continue
seen.add(url)
all_urls.append(url)
next_page = extract_next_page(result.text, current_url)
if not next_page:
break
current_url = next_page
polite_sleep()
return all_urls
Step 5: Build the dataset
Now combine the two stages.
import pandas as pd
def build_dataset(start_url: str, pages: int = 2, use_proxiesapi: bool = False) -> list[dict]:
session = requests.Session()
property_urls = crawl_results(start_url, pages=pages, use_proxiesapi=use_proxiesapi)
rows = []
for index, url in enumerate(property_urls, start=1):
result = fetch_html(session, url, use_proxiesapi=use_proxiesapi)
rows.append(parse_property_page(result.text, url))
print(f"[{index}/{len(property_urls)}] parsed {url}")
polite_sleep()
return rows
if __name__ == "__main__":
START_URL = "PASTE_YOUR_RIGHTMOVE_RESULTS_URL_HERE"
rows = build_dataset(START_URL, pages=2, use_proxiesapi=False)
pd.DataFrame(rows).to_csv("rightmove_properties.csv", index=False)
with open("rightmove_properties.json", "w", encoding="utf-8") as fh:
json.dump(rows, fh, ensure_ascii=False, indent=2)
print(f"wrote {len(rows)} records")
Where ProxiesAPI fits
You do not need a proxy for a single sanity check. You usually need one when:
- you crawl many result pages
- you revisit the same market regularly
- you collect detail pages at volume
- block rates become the bottleneck instead of parsing
With the helper above, enabling ProxiesAPI is just:
rows = build_dataset(START_URL, pages=5, use_proxiesapi=True)
That keeps the tutorial honest: the parser is still plain Python, and the proxy layer is optional until you need it.
Practical tips
- Cache HTML snapshots while developing so you do not hit the live site on every parser tweak.
- Start with one market and one or two pages before expanding.
- Log which URLs failed so retries and re-runs stay cheap.
- Expect selectors to drift over time. Keep URL discovery and field parsing separate.
Final takeaway
The winning pattern for Rightmove is not a clever one-liner. It is a boring, reliable pipeline:
- fetch politely
- discover property URLs from results pages
- parse structured fields from detail pages
- export a dataset you can refresh later
That is enough to turn Rightmove from "a website I manually browse" into "a repeatable housing data source for research, alerts, or internal tooling."
Rightmove pages are simple until you scale. ProxiesAPI gives your scraper a stable proxy layer when you need to crawl more pages without juggling IP rotation yourself.
Rightmove pages are simple until you scale. ProxiesAPI gives your scraper a stable proxy layer when you need to crawl more pages without juggling IP rotation yourself.