Robots.txt:
One Bad Line Can Kill Your Entire Site.

A robots.txt file is a plain text file placed at the root of your domain (yourdomain.com/robots.txt) that communicates crawling instructions to search engine bots. It tells crawlers which pages or directories to access and which to skip. Common uses include: blocking admin and login pages from being crawled, preventing search result or filtered pages from wasting crawl budget, keeping staging environments out of search results, and referencing the location of your XML sitemap. It is not a security tool and should not be used to hide sensitive data.

Yes — for compliant crawlers like Googlebot, Bingbot, and other major search engine bots. These crawlers respect the Robots Exclusion Protocol and follow robots.txt directives. However, robots.txt only controls crawling, not indexing. A page blocked from crawling can still appear in Google's index if other sites link to it — Google can infer the page exists from external links without being able to read its content. For true exclusion from search results, use a noindex meta tag, which controls indexing regardless of crawling status.

Yes — robots.txt remains a standard and actively used technical SEO tool in 2025. Google continues to respect and process robots.txt directives for all sites. Its role has evolved: for small business sites with a few hundred pages, crawl budget is rarely a limiting factor, so robots.txt is primarily used to block admin areas, staging paths, and duplicate content generators rather than to manage crawl budget. For large e-commerce or content sites with thousands of URLs, robots.txt is still critical for preventing crawl waste on low-value parameterized or paginated URLs.

Use robots.txt to block pages you want crawled less frequently or not at all — but that don't need to be completely hidden from search results. Appropriate uses include: blocking /wp-admin/ and other admin directories, preventing crawling of internal search result pages (/search?q=), keeping paginated or filtered category pages from using crawl budget, and excluding utility pages like /thank-you/ or /cart/. Do not use robots.txt as your only mechanism for keeping sensitive pages out of Google — pair it with noindex tags for anything that must never appear in search results.

A page 'blocked' by robots.txt means Googlebot cannot access or read its content because a Disallow directive applies to that URL. In Google Search Console's URL Inspection tool, blocked pages show as 'Crawled — currently not indexed' or 'Blocked by robots.txt.' Being blocked does not mean the URL disappears from Google's awareness — if other sites link to it, Google knows it exists. It does mean Google can't read the page's content to evaluate it for indexing.

There's no fixed crawl schedule — Google crawls each site based on its crawl budget, which is influenced by page authority, site speed, and how frequently your content changes. For most small business sites with a few dozen to a few hundred pages, Googlebot may revisit key pages every few days to a few weeks. Frequently updated content (like a news blog) gets crawled more often. You can request faster crawling of specific URLs using the URL Inspection tool in Google Search Console — this is useful after publishing new pages or making significant content updates.