n8n is a good fit for scraping when the target is a handful of pages with predictable structure, and a poor fit when you need thousands of URLs, real browser rendering, or aggressive anti-bot handling.
The community templates for this range from a three-node HTTP fetch to full browser automation driven by a model. Both extremes work. Knowing which one your problem needs, before building it, is what determines whether the workflow still runs in three months.
Table of contents
- The basic shape
- Pagination and the state problem
- Being a well-behaved client
- Where n8n stops being the right tool
- Why this workload wants to be self-hosted
- How this fits the rest of the stack
- FAQ
The basic shape
The minimum viable scraper in n8n is four nodes, and a surprising number of real use cases need nothing more.
- A trigger. A schedule for periodic collection, or a webhook if something else decides when to run.
- HTTP Request. Fetch the page. Set a realistic user agent, a timeout, and enable the response as text rather than letting it try to parse JSON.
- HTML node. Extract values with CSS selectors. This is the part that breaks when the site changes, and the part worth keeping simple.
- A destination. Write to a database, a spreadsheet, or an API. Do not let the results live only in the execution log.
Add an IF node that checks whether the extraction actually returned anything, and route the empty case to a notification. A scraper that silently returns nothing after a site redesign is worse than one that stops, because downstream systems keep consuming stale or absent data without complaint.
Pagination and the state problem
Multi-page scraping is where the workflow gets interesting, and there are three patterns depending on how the site paginates.
Known page count. Generate a list of page numbers, split into items, and let n8n run the fetch for each. Simplest by far when the total is knowable up front.
Next-link following. Extract the next-page URL from the current page and loop until it is absent. This needs a loop node and a maximum iteration count, because a site that always renders a next link will otherwise run forever.
Cursor or offset APIs. If the site has an underlying JSON API - check the network panel before writing any HTML selectors, because it very often does - paginate against that instead. It is faster, more stable, and does not break on a redesign.
For long-running collection across executions, n8n exposes static workflow data that persists between runs, which is where you store the last cursor or the highest id seen. Use it deliberately and keep it small; it is a convenience store, not a database.
Being a well-behaved client
This is the part that decides whether your scraper keeps working, and it is mostly manners.
- Rate limit yourself. A delay between requests, deliberately. Hammering a site is how you get blocked, and the block usually arrives without warning at the IP level.
- Read the terms and the robots file. Not legal advice, and worth knowing what you are ignoring if you ignore it.
- Cache aggressively. If you fetched a page an hour ago and it changes daily, do not fetch it again. This is also the cheapest performance improvement available.
- Identify yourself honestly where you can, and handle 429 responses by backing off rather than retrying immediately.
- Prefer the API. If the site offers one, use it. It is more stable, usually faster, and explicitly permitted.
Most scrapers that break do so because the site changed. Most scrapers that get blocked do so because they were impolite. The first is unavoidable and the second is entirely a choice.
Where n8n stops being the right tool
Four signals, and they are fairly bright lines.
JavaScript-rendered content. If the data is not in the raw HTML, an HTTP request will not find it. You need a headless browser, which means either an external rendering service called over HTTP or a separate scraping stack. Trying to do browser automation inside a workflow tool is possible and unpleasant.
Thousands of URLs. Workflow tools are not built for high-volume crawling. Execution history grows, memory climbs, and a single failed run loses a lot of work. At that scale you want a purpose-built framework with a proper queue and retry model.
Serious anti-bot measures. Rotating proxies, fingerprint management, and challenge solving are a specialist domain. If you are fighting that, you have left the territory where a general workflow tool helps.
Complex parsing. If extraction requires real logic rather than selectors, put it in a code node - or better, in a small service the workflow calls, where it can have tests.
The honest framing: n8n is excellent glue around scraping and a mediocre scraping engine. Use it for the trigger, the scheduling, the routing, and the destination, and delegate the hard fetching to something built for it.
Why this workload wants to be self-hosted
Scraping is the workload where hosted automation pricing hurts most, because it is high-frequency by nature. A workflow fetching two hundred pages a day is six thousand tasks a month against a per-task allowance, and that is a small scraper.
Self-hosted, that costs compute, which is close to flat regardless of how many times the workflow runs. The economics reverse completely and they reverse early.
There are two other practical advantages. Your requests come from your own IP rather than a shared platform range that may already be blocked by the target. And long-running executions are bounded by your container’s resources rather than by a platform’s execution time limit, which matters for anything paginating deeply.
How this fits the rest of the stack
A scraping workflow needs a container that stays running, a database that can absorb the results and the execution history, and enough memory for the heavy runs. The RunxBuild hosting calculator prices those pieces so the self-hosted option can be compared against per-task billing honestly. RunxBuild runs n8n as a managed tool with autoscaling, custom domains, environment variables, and logs, alongside managed MySQL and Postgres with connection limits and backups - from a $6 Basic plan.
Useful related references:
- n8n vs Make: Where the Complexity Goes
- n8n Use Cases: What It Is Genuinely Good At, and What It Is Not
- n8n Alternatives: The Honest Comparison by What You Are Escaping
- Builds on RunxBuild
FAQ
Is n8n good for web scraping?
For modest, structured extraction from pages with predictable HTML, yes - a four-node workflow covers a lot of real use cases. For thousands of URLs, JavaScript-rendered content, or sites with serious anti-bot measures, it is the wrong tool and a purpose-built scraping framework is better.
How do I handle pagination in an n8n scraper?
Three patterns: generate a list of page numbers when the count is known, follow next-page links in a loop with a hard iteration cap, or paginate against the site’s underlying JSON API if one exists. Check the network panel first - an API very often exists.
How do I scrape JavaScript-rendered pages?
Not with a plain HTTP request, since the data is not in the raw HTML. You need a headless browser, which in practice means calling an external rendering service over HTTP from the workflow, or moving the fetching into a separate scraping stack entirely.
How do I stop my scraper from getting blocked?
Rate limit yourself with a deliberate delay between requests, cache so you do not refetch unchanged pages, back off properly on 429 responses rather than retrying immediately, set a realistic user agent, and use the site’s API if it offers one.
Why self-host n8n for scraping?
Because scraping is high-frequency by nature and per-task pricing scales badly against it - two hundred pages a day is six thousand tasks a month. Self-hosted, the cost is compute and stays roughly flat. Requests also come from your own IP rather than a shared platform range.