robots.txt, Noindex and Crawl Controls

Crawling, indexing and access are different problems.

Using the wrong control can leave a URL indexed, prevent a noindex directive from being seen or expose confidential content publicly.

robots.txt

A robots.txt file tells compliant crawlers which URL paths they may request.

It is useful for managing crawler access to selected low-value or expensive paths. It is not a reliable privacy mechanism and does not guarantee that a blocked URL disappears from search.

Noindex

A noindex directive tells supporting search engines not to include the page in results.

It can be delivered through:

• A robots meta tag.
• An X-Robots-Tag HTTP header.

The crawler must be able to access the page to see the directive. Blocking the same page in robots.txt can prevent processing.

Authentication and Private Content

Confidential pages require authentication, authorization or network restrictions.

A robots file is public and can reveal the paths it mentions.

Canonicals and Removal Tools

Canonicalization selects a preferred representative among accessible duplicates. It does not prevent crawling or guarantee exclusion.

Temporary removal tools can hide a URL for a period but do not replace permanent server, indexing or access controls.

Common Mistakes

Avoid:

• Putting noindex inside robots.txt.
• Blocking a page before a crawler can see its noindex.
• Using robots.txt for private files.
• Disallowing the entire production site after launch.
• Blocking CSS or JavaScript needed for rendering.
• Conflicting plugin and server rules.

Frequently Asked Questions

Does robots.txt remove a page from Google?

Not reliably. Use the correct indexing or access control.

How do I protect confidential content?

Require authentication or another real access-control mechanism.

Continue Learning

Previous: [Canonical URLs in WordPress](/resources/wordpress-manual/seo/canonical-urls/)

Next: [How to Check Whether a WordPress Page Is Indexed](/resources/wordpress-manual/seo/check-page-indexed/)