Does Disallow remove a page from Google?

No — it only stops crawling. A disallowed URL can still be indexed if other pages link to it. Use a noindex tag (on a crawlable page) to keep it out of results.

Where must robots.txt live?

At the root of the host, e.g. https://example.com/robots.txt. Each subdomain needs its own file.

Free Robots.txt Tester

Fetch, parse and test robots.txt rules for any crawler.

Create a free account for more

Audit your whole site (not just one page), track keyword rankings, schedule weekly checks and get an AI fix plan.

Start free

What does the Robots.txt Tester do?

Load a site’s robots.txt (or paste your draft), see it syntax-highlighted with warnings, and test whether a specific path is allowed for Googlebot, Bingbot or any other user agent. The matching follows RFC 9309: the longest matching rule wins and Allow wins ties.

How to use it

1Enter a site URL or paste robots.txt content.
2Type the path you want to test and choose a user agent.
3Read the verdict and the exact rule (highlighted) that decided it.

Frequently asked questions

Does Disallow remove a page from Google?: No — it only stops crawling. A disallowed URL can still be indexed if other pages link to it. Use a noindex tag (on a crawlable page) to keep it out of results.
Where must robots.txt live?: At the root of the host, e.g. https://example.com/robots.txt. Each subdomain needs its own file.

Start growing your organic traffic today

Create a free account in under a minute. No credit card required — upgrade only when you need more.

Start free See pricing