Related: Google Sheets as a Database: When It Works and When It Breaks
This article is general information, not legal advice. The law differs between countries and is still developing. For a real project, especially one involving personal data or a large commercial use, talk to a lawyer in your jurisdiction.
"Is scraping legal?" has no single answer. Scraping is a technique, like photocopying. Whether a particular use is lawful depends on what you collect, how you collect it, and what you do with it afterwards.
Here are the questions that decide it.
1. Is the data public or behind a login?
This is the biggest single factor.
- Public pages anyone can view without an account are the lowest risk category.
- Data behind a login is riskier, because creating an account usually means accepting terms that forbid scraping.
- Getting around a barrier, such as defeating a login, a paywall, a CAPTCHA or an IP block, is where the serious legal risk starts.
In the United States, the main computer misuse law is the Computer Fraud and Abuse Act. The Supreme Court's 2021 decision in Van Buren v. United States narrowed it, focusing on whether you were allowed through the gate at all. In hiQ Labs v. LinkedIn, a federal appeals court found that scraping publicly available profiles was unlikely to breach that law. Courts elsewhere work under different laws, so this is not a worldwide rule.
2. Did you agree to terms that forbid it?
Even where scraping is not a crime, it can be a breach of contract.
The hiQ case shows this well. hiQ largely won the computer misuse argument. It later lost on contract grounds, because it had agreed to LinkedIn's user agreement. In 2024, Meta's case against the data company Bright Data went the other way: the court found Meta had not shown that its terms prohibited logged-out scraping of public data, and Meta dropped the claims.
The practical point: an account you signed up for, with terms you clicked to accept, is a much stronger contract against you than a terms page you never saw.
In the EU, courts have also held that a website's terms can restrict how its data is used, even where the database itself has no special legal protection.
3. Is any of it personal data?
This is where many projects get into trouble.
Under the GDPR in Europe, and similar laws in many other countries, personal data is protected even when it is publicly visible. Names, email addresses, profile photos, phone numbers and usernames all count. "It was on a public page" is not a lawful basis on its own.
If you collect personal data you generally need:
- A lawful basis for processing it
- To collect only what you actually need
- To tell people you have it, in many cases
- To keep it secure and delete it when you are done
Regulators do enforce this against scrapers. Clearview AI has been fined by several European data protection regulators for scraping publicly available photos.
The lowest risk path is simple: do not collect personal data unless the project genuinely needs it.
4. Are you copying protected content?
Collecting facts and republishing creative work are very different things.
- Facts are generally not protected by copyright. Prices, opening hours, product specifications and stock levels are usually facts.
- Creative expression is protected. Articles, reviews, photographs and product descriptions written by someone are usually protected.
- In the EU, databases have their own right. A database built with substantial investment can be protected even when the individual facts in it are not. Extracting a substantial part of it can infringe.
Analysing content internally is lower risk than publishing it. Copying someone's product descriptions onto your own site is not analysis. It is copying.
5. Are you harming the site?
Sending thousands of requests a second can slow or crash a site. Beyond being rude, heavy load can support legal claims, and it gets your traffic blocked quickly.
- Rate limit your requests
- Scrape at quiet times
- Cache what you have already fetched instead of fetching it again
- Stop immediately if you are asked to
What about robots.txt?
A robots.txt file tells automated visitors which parts of a site they may crawl. In most places it is not a law in itself. Ignoring it still weakens your position in any dispute, and it is plainly poor practice. Respect it.
A practical checklist
- Check for an official API or data feed first. It is more reliable, and the permission question is already answered.
- Read the site's terms, especially if you need an account.
- Stick to public pages. Never bypass logins, paywalls, CAPTCHAs or blocks.
- Respect robots.txt.
- Identify your bot honestly in its user agent, with a way to contact you.
- Rate limit and scrape off peak.
- Avoid personal data, or collect the minimum and handle it under privacy law.
- Do not republish creative content. Use facts. Link to sources.
- Keep only what you need, and for as long as you need it.
- Get legal advice for anything large, commercial, or involving people's data.
For how to build a scraper that is reliable as well as polite, see web scraping architecture. If you are planning a data project and want it built carefully, talk to us.
Comments