Web Scraping with R: rvest + httr2 Tutorial
If you search for web scraping with R, most examples stop at a toy page and a single CSS selector.
That is not enough for real work.
A practical R scraping workflow needs three things:
- a request layer you can control
- a parser that survives imperfect markup
- an export path into tidy data frames
The cleanest stack for that today is:
httr2for HTTP requestsrvestfor HTML parsingdplyr,purrr, andreadrfor shaping output
R is great for analysis-first scraping workflows. When your targets start rate-limiting or fingerprinting aggressively, ProxiesAPI can sit in front of the same parser instead of forcing a rewrite.
Why httr2 + rvest is the right combination
rvest is excellent at extracting nodes and text, but it is not meant to be your full request orchestration layer.
httr2 gives you:
- explicit headers
- retry control
- request body tools
- response inspection
rvest gives you:
- CSS selectors
- table extraction
- link extraction
- predictable HTML traversal
Put simply:
| Job | Best package |
|---|---|
| Fetch the page | httr2 |
| Parse HTML nodes | rvest |
| Shape the dataset | dplyr / purrr |
| Save results | readr |
That split keeps your code readable.
Install the packages
install.packages(c("httr2", "rvest", "xml2", "dplyr", "purrr", "readr", "stringr", "tibble"))
Then load them:
library(httr2)
library(rvest)
library(dplyr)
library(purrr)
library(readr)
library(stringr)
library(tibble)
A realistic example: scrape Hacker News front-page stories
Hacker News is a good tutorial target because:
- the page is public
- the HTML is server-rendered
- selectors are easy to verify
- you can immediately turn the result into a tibble
library(httr2)
library(rvest)
library(dplyr)
library(purrr)
library(stringr)
library(tibble)
url <- "https://news.ycombinator.com/"
resp <- request(url) |>
req_user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/137.0.0.0 Safari/537.36") |>
req_perform()
html <- resp |>
resp_body_html()
Now parse the story rows:
story_rows <- html |> html_elements("tr.athing")
stories <- map_dfr(story_rows, function(row) {
story_id <- row |> html_attr("id")
title_node <- row |> html_element("span.titleline > a")
subtext_row <- row |> xml2::xml_find_first("following-sibling::tr[1]")
score_text <- subtext_row |> html_element("span.score") |> html_text2()
author <- subtext_row |> html_element("a.hnuser") |> html_text2()
age <- subtext_row |> html_element("span.age a") |> html_text2()
links <- subtext_row |> html_elements("a")
comments_text <- if (length(links) > 0) html_text2(links[[length(links)]]) else NA_character_
tibble(
id = story_id,
title = title_node |> html_text2(),
url = title_node |> html_attr("href"),
points = str_extract(score_text, "\\\\d+") |> as.integer(),
author = author,
age = age,
comments = str_extract(comments_text, "\\\\d+") |> as.integer()
)
})
stories
That is already a legitimate R scraping workflow, not a toy one-liner.
Extract links, attributes, and tables
The three patterns you will use most often are:
1. Text from repeated cards
html |> html_elements(".product-card .title") |> html_text2()
2. Attributes like URLs or image sources
html |> html_elements(".product-card a") |> html_attr("href")
html |> html_elements(".product-card img") |> html_attr("src")
3. HTML tables
tables <- html |> html_elements("table") |> html_table()
If a site already exposes the data in a table, html_table() is often the fastest win.
Pagination in R
Real scrapers rarely stop at one page. Here is a simple pagination pattern:
pages <- 1:5
fetch_hn_page <- function(page_num) {
page_url <- if (page_num == 1) {
"https://news.ycombinator.com/"
} else {
paste0("https://news.ycombinator.com/?p=", page_num)
}
request(page_url) |>
req_user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/137.0.0.0 Safari/537.36") |>
req_perform() |>
resp_body_html()
}
all_stories <- map_dfr(pages, function(page_num) {
html <- fetch_hn_page(page_num)
rows <- html |> html_elements("tr.athing")
map_dfr(rows, function(row) {
tibble(
page = page_num,
id = row |> html_attr("id"),
title = row |> html_element("span.titleline > a") |> html_text2()
)
})
})
That pattern generalizes well: define a page builder, fetch each page, extract the repeated node, bind the results.
When R is a great choice for scraping
R is a strong fit when:
| Situation | Why R works well |
|---|---|
| Analysis is the main goal | You can scrape, clean, chart, and model in one environment |
| The pages are mostly HTML | rvest handles CSS-selector extraction cleanly |
| You need CSV, Parquet, or tibble output fast | Tidyverse pipelines stay concise |
| The audience is analysts, not backend engineers | The code reads closer to data manipulation than app plumbing |
R is a weaker fit when:
| Situation | Better choice |
|---|---|
| Heavy browser automation | Playwright or Selenium |
| Large async crawling systems | Python, Node.js, or Go |
| Deep anti-bot evasion work | A dedicated fetch layer plus browser tooling |
That does not mean R is bad at scraping. It means you should use it where it has leverage.
Add retries and better request hygiene
Do not hammer pages with the default request every time. At minimum:
safe_fetch <- function(url) {
request(url) |>
req_user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/137.0.0.0 Safari/537.36") |>
req_headers(
"Accept-Language" = "en-US,en;q=0.9"
) |>
req_retry(max_tries = 4) |>
req_perform()
}
That small amount of request hygiene prevents a surprising number of flaky tutorial failures.
How ProxiesAPI fits into an R workflow
You do not need to change the parsing code. Just wrap the target URL before req_perform().
proxiesapi_url <- function(target_url, api_key) {
paste0(
"https://api.proxiesapi.com/?auth_key=",
utils::URLencode(api_key, reserved = TRUE),
"&url=",
utils::URLencode(target_url, reserved = TRUE)
)
}
resp <- request(proxiesapi_url("https://example.com", Sys.getenv("PROXIESAPI_KEY"))) |>
req_user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/137.0.0.0 Safari/537.36") |>
req_perform()
That separation matters. It lets you keep the data-cleaning work in R while moving fetch complexity elsewhere.
Export the results
write_csv(stories, "hn_stories.csv")
saveRDS(stories, "hn_stories.rds")
From there you can move directly into:
ggplot2charts- clustering
- time-series comparisons
- R Markdown reports
That is the real advantage of web scraping with R: the handoff from HTML to analysis is almost frictionless.
Bottom line
If your pages are mostly static HTML and your end goal is analysis, httr2 + rvest is the practical default.
Use:
httr2to fetch pages intentionallyrvestto parse nodes cleanly- tidyverse tools to shape the output
Then only upgrade the transport layer when anti-bot complexity actually appears. That keeps your R pipeline simple, readable, and easy to maintain.
R is great for analysis-first scraping workflows. When your targets start rate-limiting or fingerprinting aggressively, ProxiesAPI can sit in front of the same parser instead of forcing a rewrite.