Can Robots.txt block a page from Google search results?

No. Robots.txt can limit crawling, but a blocked URL may still appear in search results if Google discovers the address through other signals.

Where should a Robots.txt file be placed?

A Robots.txt file belongs at the root of the relevant host, such as the main domain or a subdomain. Crawlers request that root location directly.

Does Robots.txt affect website visitors?

Robots.txt normally guides automated crawlers, not human visitors. People can usually open a disallowed URL unless separate access controls restrict the page.

Glossary

What Is Robots.txt?

Robots.txt is a plain-text file that gives search engine crawlers instructions about which website paths they may request. The file sits at the website’s root address and can allow or disallow access for specified crawler user agents. Robots.txt guides crawling, but it does not guarantee that blocked URLs stay out of search results.

Quick Facts About Robots.txt

Key Takeaways About Robots.txt

It controls crawling access, not guaranteed search engine indexing.
The file must be available at the website’s root address.
Robots.txt is not the same as a noindex tag.

How Robots.txt Works

A hand holds a pen over a paper titled "robots.txt" beside a laptop, coffee mug, mouse, and notebook on a desk

Robots.txt contains directives that crawlers read before requesting website paths. A rule can apply to a named user agent or to all crawlers, then allow or disallow selected paths. The file works at the root of a host, so a separate file may apply to a different subdomain.

Related glossary terms: Crawl Budget, Crawl Error, Noindex Tag.

Search engines may follow the instructions when deciding which URLs to crawl. Crawl access and indexing are separate decisions, so a blocked URL can still become known through links or other signals. A noindex tag gives a different instruction because it addresses whether a page should appear in search results after a crawler can access it.

Practical distinction

Robots.txt suits crawl management for areas such as internal search results or repeated parameter URLs. The rules should not block resources that search engines need to understand a page, including relevant scripts or stylesheets.

Hands write on a clipboard marked robots.txt beside a laptop, coffee mug, and SEO services note on a desk

Expert Note

When we review Robots.txt, we check its location, syntax, user-agent rules, and disallowed paths. We compare those rules with the website’s important pages and crawl data. Our team also checks whether blocked resources affect how search engines render the site.

Robots.txt in Practice: A Real-World Example

A Ballarat service website may use Robots.txt to prevent crawlers from requesting internal search result URLs. The file can leave public service pages accessible while reducing repeated, low-value crawl paths.