Web Scraping with Scrapy: Getting Started Guide

Scrapy is what you graduate to when requests + BeautifulSoup stops being enough.

If you only scrape one page, plain Python scripts are fine. If you need retries, pagination, concurrency, exports, and something you can run again next month without hating yourself, Scrapy is a much better fit.

This getting started guide walks through the pieces that matter most:

• project setup
• a real spider
• CSS and XPath selectors
• pagination
• item pipelines
• exports
• adding ProxiesAPI for proxy rotation

When to use Scrapy

Scrapy shines when the job is bigger than one script.

Use Scrapy when:

• you need to crawl many pages
• you want structured project code
• you need queueing, throttling, and retries
• you want exports built in

Skip it when a browser is mandatory for every page. In that case, Playwright is usually the better first choice.

Step 1: Create the project

python3 -m venv .venv
source .venv/bin/activate
pip install scrapy python-dotenv

scrapy startproject quotes_crawler
cd quotes_crawler

You will get a structure like:

quotes_crawler/
  scrapy.cfg
  quotes_crawler/
    items.py
    middlewares.py
    pipelines.py
    settings.py
    spiders/

For a first project, that is more than enough.

Step 2: Write your first spider

We will use https://quotes.toscrape.com/, a classic practice site with pagination and detail-free listing pages.

Create quotes_crawler/spiders/quotes.py:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
                "source_url": response.url,
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Run it:

scrapy crawl quotes -O quotes.json

That command alone teaches the main Scrapy loop:

• parse a page
• yield items
• follow the next link

Step 3: Understand selectors

Scrapy supports CSS and XPath. Start with CSS unless you have a clear reason not to.

Examples:

response.css("h1::text").get()
response.css("a::attr(href)").getall()
response.xpath("//div[@class='quote']/span[@class='text']/text()").get()

Rules of thumb:

• CSS is shorter and easier to read
• XPath is better for complex relationships
• inspect one page carefully before you write broad selectors

If you ever get empty data, the bug is usually your selector, not Scrapy.

Step 4: Follow detail pages

Real crawls usually need two-step extraction: listing page first, detail page second.

Here is the same spider pattern on Books to Scrape:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for href in response.css("article.product_pod h3 a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_book)

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_book(self, response):
        yield {
            "title": response.css("div.product_main h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "availability": " ".join(
                value.strip()
                for value in response.css("p.availability ::text").getall()
                if value.strip()
            ),
            "url": response.url,
        }

This pattern is the backbone of most production spiders.

Step 5: Use Items and Pipelines

As projects grow, anonymous dictionaries turn messy fast. Define an item and clean it in a pipeline.

items.py

import scrapy


class BookItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    availability = scrapy.Field()
    url = scrapy.Field()

pipelines.py

import re


class CleanTextPipeline:
    def process_item(self, item, spider):
        if item.get("price"):
            item["price"] = re.sub(r"\s+", " ", item["price"]).strip()
        if item.get("availability"):
            item["availability"] = re.sub(r"\s+", " ", item["availability"]).strip()
        return item

Enable it in settings.py:

ITEM_PIPELINES = {
    "quotes_crawler.pipelines.CleanTextPipeline": 300,
}

That keeps cleanup logic out of the spider.

Step 6: Add practical crawler settings

New users often miss this step. Good settings matter as much as the spider.

In settings.py:

ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 0.75
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS = 8
RETRY_TIMES = 3
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
FEED_EXPORT_ENCODING = "utf-8"

Why these matter:

• DOWNLOAD_DELAY reduces accidental hammering
• AUTOTHROTTLE_ENABLED smooths traffic automatically
• RETRY_TIMES handles occasional failures
• FEED_EXPORT_ENCODING prevents ugly CSV and JSON text issues

Step 7: Export data

Scrapy makes exports pleasantly boring:

scrapy crawl quotes -O quotes.json
scrapy crawl quotes -O quotes.csv
scrapy crawl books -O books.jl

For many teams, that is enough. You do not need a database on day one. A clean JSON or CSV file is already a useful deliverable.

Step 8: Add ProxiesAPI to Scrapy

When you scale to tougher targets, proxy support belongs in middleware or request settings, not inside every parser.

In middlewares.py:

import os


class ProxiesApiMiddleware:
    def process_request(self, request, spider):
        key = os.getenv("PROXIESAPI_KEY")
        if not key:
            return None

        proxy = f"http://{key}:@proxy.proxiesapi.com:10000"
        request.meta["proxy"] = proxy
        return None

Enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "quotes_crawler.middlewares.ProxiesApiMiddleware": 350,
}

That is the clean integration point. Your spiders stay focused on extraction while the network layer handles routing.

Common mistakes

• Writing selectors before inspecting the page structure.
• Mixing parsing logic with cleanup logic.
• Running high concurrency too early.
• Using a browser when the site is just HTML.
• Ignoring retries and timeouts until the crawl fails in production.

The fix is almost always better separation of concerns:

• spider for discovery
• pipeline for cleanup
• settings and middleware for network behavior

Final takeaway

Scrapy is worth learning because it turns scraping from "a script that works today" into "a crawler I can operate."

If you learn four concepts early, you are already ahead:

1. selectors
2. pagination
3. items and pipelines
4. network settings

Once those are in place, adding exports, retries, and ProxiesAPI is straightforward instead of painful.