Is web scraping legal? What the law and site terms usually say

Short answer: collecting publicly available web pages is not automatically illegal, and it is not automatically safe either. The answer depends on the country, what you collect, how you collect it, what the site's terms say, and what you do with the data. This is general information, not legal advice. For a commercial project, ask a lawyer who knows your jurisdiction.

Which laws can apply?

Computer misuse laws

In the United States, the Computer Fraud and Abuse Act covers access "without authorization". In the hiQ v. LinkedIn litigation, the Ninth Circuit found that scraping publicly accessible profile pages was unlikely to violate that act. That does not settle every case, and it did not end the dispute, which later turned on contract grounds. Bypassing a login, a paywall or technical blocks is on much weaker ground. India's Information Technology Act has provisions on unauthorised access, and the UK and EU have similar laws.

Contract and terms of service

A site's terms may forbid automated access. If you agreed to them by creating an account, they can bind you, and a breach can lead to account bans or a civil claim. Terms matter less for pages you reach without any agreement, but the position varies by country.

Copyright and database rights

Facts are generally not protected by copyright. Creative text, photos and the selection and arrangement of a database can be. Copying and republishing articles or images is an infringement risk. The EU also has a separate database right. Whether using scraped material to train a model is allowed is being argued in courts and legislatures now, and the answer differs by country.

Data protection

If you collect information about identifiable people, such as names, emails or profile data, laws like the GDPR in Europe and India's Digital Personal Data Protection Act can apply, even if the data was public. You need a lawful basis, a purpose, limits on retention and a way to honour deletion requests.

What does robots.txt do?

robots.txt is a convention that tells crawlers which paths they should not fetch. It is not a law, and ignoring it is not itself a crime in most places. It is still the clearest statement of what the site owner wants, and courts and regulators look at whether you respected clear signals. Follow it.

What are sensible rules to follow?

• Check for an official API or data export first. It is faster, more stable and clearly allowed.
• Read the terms and robots.txt for the site.
• Stay on public pages. Do not log in with accounts you are not allowed to automate, and do not bypass technical protections like CAPTCHAs or IP blocks.
• Be gentle: identify your bot with a real User-Agent and contact address, limit requests per second, cache, and back off on errors. Hammering a small site can cause real harm and turns a grey area into a clear problem.
• Collect only what you need. Avoid personal data unless you have a lawful reason and a plan for it.
• Do not republish copyrighted content. Extract facts, link back, and summarise instead of copying.
• Keep records of what you collected, when and why.

What about price monitoring or research?

Collecting public prices or listings for analysis is common and generally lower risk, provided you are polite and do not circumvent protections. Research use sometimes has additional exceptions. Commercial resale of someone else's database is where claims appear.

What if you receive a cease and desist?

Stop, keep the evidence, do not delete anything yet, and talk to a lawyer. Do not argue with the sender by email.