Web Scraping with Ruby: Nokogiri + HTTParty Tutorial

If you are searching for web scraping with Ruby, the best beginner-to-production path is still very simple:

• use HTTParty to fetch pages
• use Nokogiri to parse HTML
• add retries, delays, and export logic yourself

That stack is not flashy, but it is dependable.

In this tutorial we will build a Ruby scraper that:

• fetches listing pages
• parses repeated product cards
• follows pagination
• retries transient failures
• exports a CSV you can actually use

We will use books.toscrape.com as the target because it is public, stable, and ideal for learning selectors.

Why Nokogiri + HTTParty is still the default answer

Ruby has multiple scraping libraries, but this pairing is hard to beat for clarity.

The point is not to chase the biggest framework. The point is to build a scraper you can read six months later.

Setup

Create a small project:

mkdir ruby-scraper
cd ruby-scraper
bundle init
bundle add httparty nokogiri retryable

You could write your own retry loop, but retryable keeps the code tidy.

What we are scraping

We will scrape the catalog listing pages on:

https://books.toscrape.com/catalogue/page-1.html

Each product card contains:

• title
• price
• stock text
• product detail URL

The repeated item container is:

• article.product_pod

Inside each card we care about selectors like:

• h3 a
• p.price_color
• p.instock.availability

That is the core pattern for most Ruby scraping work: find the repeated container, then parse fields inside it.

Step 1: Build the fetcher

require "httparty"
require "nokogiri"
require "csv"
require "retryable"
require "uri"

class Fetcher
  USER_AGENT = "Mozilla/5.0 (compatible; ProxiesAPI-Guides/1.0)"
  TIMEOUT = 30

  def initialize(proxiesapi_key: nil)
    @proxiesapi_key = proxiesapi_key
  end

  def get(url)
    target = @proxiesapi_key ? proxiesapi_url(url) : url

    Retryable.retryable(tries: 4, sleep: lambda { |n| 2**n }) do
      response = HTTParty.get(
        target,
        timeout: TIMEOUT,
        headers: {
          "User-Agent" => USER_AGENT,
          "Accept-Language" => "en-US,en;q=0.9"
        }
      )

      raise "HTTP #{response.code}" if response.code >= 400
      body = response.body.to_s
      raise "short body" if body.length < 1_000
      body
    end
  end

  private

  def proxiesapi_url(url)
    "https://api.proxiesapi.com/?auth_key=#{URI.encode_www_form_component(@proxiesapi_key)}&url=#{URI.encode_www_form_component(url)}"
  end
end

That is already most of the network discipline you need for a lot of real scrapers.

Step 2: Parse one page with Nokogiri

class BookPageParser
  BASE_URL = "https://books.toscrape.com/"

  def parse_listing(html)
    doc = Nokogiri::HTML(html)

    doc.css("article.product_pod").map do |card|
      link = card.at_css("h3 a")
      price = card.at_css("p.price_color")&.text&.strip
      availability = card.at_css("p.instock.availability")&.text&.strip

      {
        title: link&.[]("title") || link&.text&.strip,
        url: absolute_url(link&.[]("href")),
        price: price,
        availability: availability
      }
    end
  end

  def next_page(html)
    doc = Nokogiri::HTML(html)
    href = doc.at_css("li.next a")&.[]("href")
    return nil unless href

    absolute_url("catalogue/#{href}")
  end

  private

  def absolute_url(path)
    return nil if path.nil? || path.empty?
    URI.join(BASE_URL, path).to_s
  end
end

This is the real value of Nokogiri: CSS selectors stay readable, and the parsing code stays compact.

Step 3: Crawl multiple pages

class CatalogScraper
  def initialize(fetcher:, parser:)
    @fetcher = fetcher
    @parser = parser
  end

  def scrape(start_url:, max_pages: 5)
    url = start_url
    page_num = 0
    rows = []
    seen = {}

    while url && page_num < max_pages
      page_num += 1
      html = @fetcher.get(url)
      batch = @parser.parse_listing(html)

      batch.each do |row|
        next if seen[row[:url]]
        seen[row[:url]] = true
        rows << row
      end

      url = @parser.next_page(html)
      sleep(rand * 1.2 + 0.4)
    end

    rows
  end
end

This structure scales nicely because responsibilities stay separate:

• Fetcher gets HTML
• BookPageParser extracts fields
• CatalogScraper controls crawl flow

That separation matters more than clever code golf.

Step 4: Export CSV

def export_csv(rows, path)
  CSV.open(path, "w") do |csv|
    csv << %w[title price availability url]
    rows.each do |row|
      csv << [row[:title], row[:price], row[:availability], row[:url]]
    end
  end
end

fetcher = Fetcher.new(proxiesapi_key: ENV["PROXIESAPI_KEY"])
parser = BookPageParser.new
scraper = CatalogScraper.new(fetcher: fetcher, parser: parser)

rows = scraper.scrape(
  start_url: "https://books.toscrape.com/catalogue/page-1.html",
  max_pages: 3
)

export_csv(rows, "books.csv")
puts "scraped #{rows.length} rows"

If you open the CSV, you now have a clean small dataset with price and availability text.

Practical Ruby scraping advice

If you are serious about web scraping with Ruby, these habits matter more than which gem is coolest:

Most “my scraper broke” stories come from skipping these basics, not from picking the wrong parser.

When to use Ruby and when not to

Ruby is a good choice when you want:

• highly readable scripts
• a fast path from idea to working crawler
• good text-processing ergonomics

Ruby is a weaker fit when you need:

• massive concurrency from one process
• deep browser automation as the primary path
• a team that already standardizes on Python or Node

For server-rendered HTML targets, though, Ruby is absolutely good enough.

Where ProxiesAPI fits

At very small scale, nowhere. Just fetch directly.

It becomes useful when you:

• scrape many pages repeatedly
• move from one demo target to many real targets
• start seeing 429s, captchas, or unstable response quality

That is why the fetcher accepts a proxiesapi_key but does not force it. The parser should not care whether the HTML came directly or through a proxy layer.

Final thoughts

The right answer to web scraping with Ruby is usually not a giant framework. It is a boring, maintainable script built from:

• HTTParty
• Nokogiri
• retries
• export

That stack teaches the right instincts: fetch carefully, parse predictable containers, keep responsibilities separate, and only add more machinery when the target actually demands it.

If you can build one clean scraper this way, you can reuse the same pattern across catalogs, directories, blogs, job boards, and a surprising number of internal-data tasks.