9 min read
How Web Scraping Works and Whether It's Legal
Scraping is surrounded by two myths: that it's “hacking”, and that it's “just copying someone's site”. It's neither. Here's how it actually works, where the legal line runs, and what part of it a business genuinely needs.
A scraper is a program that opens a page exactly like your browser does and takes the data you need from it: product name, price, availability, a contact from a public directory. The only difference is that a person does this with their eyes and hands, while a program does it automatically, a thousand times over.
There's no “hacking” involved: a scraper sees precisely what any visitor to the site sees. If a page isn't visible without a password, the scraper doesn't see it either. That's why the legal question is almost never about the technology. It's about WHICH data you take and WHAT you do with it afterwards.
How it works, step by step
- The program opens a page by its address — the same way a browser does.
- It finds the relevant spots in the page's code: the price block, the product title, the phone number on a listing.
- It writes what it found into a spreadsheet or database — structured, in tidy columns.
- It repeats this for the next pages: categories, product cards, pagination.
- On a schedule it runs the whole thing again — daily, hourly, whatever we agree — and shows exactly what changed since last time.
Is it legal?
The short answer: collecting public data isn't prohibited as such, but legality depends on three things — which data it is, where it comes from, and how you use it. The very same technology can be entirely lawful or entirely unlawful depending on those answers.
One thing up front: I'm a developer, not a lawyer, and this article explains practice rather than giving legal advice. If the project is large or the task falls into a grey area, that's a question for a lawyer — and I'll be the first to say so, rather than take the money and stay quiet.
Three zones: safe, careful, and no
Public data for your own analysis
Green zonePrices, stock levels, product specs, open catalogues — what any visitor sees without a password. You collect it to make decisions inside your company: compare prices, spot shortages, build market analytics. The most common and most comfortable scenario.
Personal data and someone else's rules
Amber zoneNames, phone numbers, emails are personal data, and separate legislation applies to them: you need a lawful basis for processing, not merely the technical ability to collect. The same goes for the source site's terms of use, which may explicitly forbid automated collection. Every such case gets considered on its own.
Bypassing protection and republishing
Red zoneGuessing passwords, defeating CAPTCHAs, reaching restricted areas — and copying someone's content to publish it as your own. That isn't scraping any more, it's an offence, from copyright infringement to unauthorised access. I don't take that work.
The rules I build scrapers by
- Only what's reachable without logging in. No borrowed passwords, no defeating protection.
- Considerate speed. The scraper works so it doesn't load the source site: paced requests, not an avalanche.
- robots.txt gets read. It isn't law, it's the site owner's request — and it deserves respect. If a section is explicitly closed, we don't go there.
- The data goes to you, not “into the open”. We collect for your analytics, not to make a copy of somebody's site.
- If a task feels doubtful, I say so before the work starts — not after the invoice.
What a scraper costs
Single-source scraper
from $400 · 1–2 weeksOne site, the fields you need, exported to a spreadsheet or Telegram. Right when you want to watch one competitor's prices or pull a catalogue from a single marketplace.
More about this serviceSeveral sources with reports
from $650Three to five sites normalised into one format, plus a regular report: what changed, where the price dropped, what went out of stock. The most frequent request in retail.
More about this serviceMaintenance and updates
$100 per monthThe part nobody warns you about: source sites change their markup, and every scraper breaks sooner or later. The subscription is what keeps it running instead of quietly dying a month in.
What to ask yourself before starting
Three questions clear up most of the doubt. First: can an ordinary visitor see this data without a password? Second: does it include personal data — names, phone numbers, emails? Third: what will you do with the result — analyse it internally or publish it outward?
If the answers are “yes, no, analyse it” — that's the green zone and the task is entirely routine. A “yes” to the second or “publish” to the third doesn't mean “you can't”; it means “this needs looking at separately”. Describe the task and I'll tell you honestly what to do with it.
Frequently asked questions about scraping
Technically the site owner sees the requests, just as they see any visitor. That's exactly why a scraper should behave calmly: moderate speed, pauses, sensible volume. Aggressive collection creates load, and blocking is a fair response to it — one more reason not to chase speed.
It's a question of when, not if: everyone changes their markup eventually. Fixing those breakages is what maintenance covers — which is why it's a separate line item. Without it the scraper still runs, but one day it quietly stops, and that tends to go unnoticed for a while.
Technically yes; legally these are personal data, and the mere fact that they're published somewhere doesn't grant an automatic right to email them. So I take such tasks only when the lawful basis is clear, and I say plainly when I can't see one.
Their public product pages are technically reachable, but big platforms usually forbid automated collection in their terms outright and defend against it seriously. Often the smarter route is the official one — many of them offer a partner API. It's more reliable and cheaper to maintain.
Need data that nobody gives you in a usable form?
Tell me which sites and exactly what needs collecting — I'll say whether it's feasible technically, comfortable rule-wise, and what it would cost.