What Is Robots.txt? Definition and Why It Matters

Robots.txt is a plain text file placed at the root of a site — reachable at yoursite.com/robots.txt — that gives crawlers instructions about which paths they are and aren’t allowed to crawl. It’s one of the oldest and simplest technical SEO mechanisms still in daily use, and also one of the most commonly misunderstood.

What robots.txt actually does

• Stops compliant crawlers from requesting and crawling specified paths or file types.
• Can point crawlers to your XML sitemap, making discovery more efficient (XML sitemap).
• Helps large sites avoid wasting crawl budget on low-value URL patterns, such as internal search results or admin paths.

What robots.txt doesn’t do

• It doesn’t guarantee a page stays out of search results. A URL blocked from crawling can still appear in search if it’s linked to from elsewhere — search engines may index the URL itself without ever crawling its content. Use noindex if you actually want a page excluded from results.
• It doesn’t secure anything. The file is publicly readable by anyone, and non-compliant bots ignore its instructions entirely, so it should never be used to hide sensitive content.
• It doesn’t remove a page already indexed — that requires either noindex or a formal removal request.

Common mistakes

• Accidentally blocking CSS or JavaScript files that pages need to render properly, which can make search engines see a broken or incomplete version of the page.
• Leaving a blanket Disallow: / rule in place after a staging site goes live, silently blocking the entire production site from crawling.
• Blocking pages via robots.txt that you’ve also noindexed — since the crawler can’t see the noindex tag if it never crawls the page, the two directives conflict and the noindex never takes effect.
• Assuming robots.txt is a substitute for access control on sensitive or private pages.

A worked example

A site launches a new staging environment to test a redesign, protected by a robots.txt rule blocking all crawling. Months later, the redesign goes live on the production domain, and someone copies the old robots.txt file across without checking its contents — inadvertently blocking the entire live site from being crawled at all. Traffic quietly drops over the following weeks as pages fall out of the index, and the cause traces back to a single leftover disallow rule. This is a surprisingly common real-world failure mode after site migrations and relaunches.

Checking robots.txt during link building

When checking a guest post placement, make sure the host site’s robots.txt doesn’t block the specific path your post lives on — some sites unintentionally (or intentionally) exclude entire blog sections from crawling (live-post checklist).

Related terms

Noindex, crawl budget, XML sitemap.

Testing robots.txt changes before publishing them

Because a single misplaced rule can block far more than intended, test any robots.txt change with a crawling tool or a search engine’s own testing feature before publishing it live. Many platforms also cache robots.txt for a period after it’s fetched, so a fix made in a panic may not take effect as quickly as expected — plan for that delay rather than assuming an edit resolves the problem instantly.

Robots.txt for subdomains

Each subdomain needs its own robots.txt file; rules set on the main domain don’t automatically apply to a blog or shop hosted on a subdomain. Sites that move their blog to a subdomain sometimes forget this and end up with an entirely unprotected or, conversely, an entirely blocked section without realising the root cause.

Robots.txt and multiple crawlers

Different search engines and AI crawlers each identify themselves with their own user-agent string, and robots.txt rules can target them individually rather than only offering blanket rules for every crawler at once. Sites wanting to allow standard search crawling while restricting other automated bots increasingly rely on this per-crawler targeting rather than a single one-size-fits-all rule.

Version history and change tracking

Because robots.txt changes can have outsized, site-wide consequences, keep a simple changelog of edits — even a basic dated note of what changed and why — so that if traffic drops unexpectedly, checking recent robots.txt history is one of the first, fastest diagnostic steps rather than a forgotten possibility.

When in doubt about a specific rule’s effect, test it against real example URLs from your site using a crawler or testing tool rather than reasoning about the syntax in the abstract, since small pattern-matching mistakes are easy to make and easy to miss by eye.

Keep the file itself as simple as possible — a handful of clear, well-tested rules is far easier to reason about and safely maintain than a long list of narrow, overlapping exceptions accumulated over years. When ownership of the file changes hands within a team, make sure whoever inherits it understands exactly what each rule is protecting against, not just what it technically does.

Where to go next

Robots.txt review belongs in every technical checklist before a link campaign, and it’s worth spot-checking on the marketplace publishers you’re considering, too.

Browse every term in the link building glossary.

Frequently asked questions