Robots.txt

Robots.txt is a file at a site's root that tells well-behaved crawlers which parts of the site they should and shouldn't crawl.

Blocking a URL in robots.txt prevents crawling but doesn't guarantee it stays out of the index — if other pages link to a blocked URL, Google can still index the URL itself (usually with no content shown, just the bare URL) based on those external signals. To reliably keep a page out of search results entirely, use a noindex tag on the page itself (which requires the page to be crawlable so the tag can be seen) rather than blocking it in robots.txt.

Robots.txt is also purely advisory for well-behaved crawlers — it does nothing to prevent access by crawlers or scrapers that simply ignore it, so it should never be relied on as a security or access-control mechanism.