Robots.txt Mistakes That Block Google and AI Crawlers

In shortOpen yourdomain.com/robots.txt right now. If you see “Disallow: /” under “User-agent: *”, or rules blocking /wp-content/ or your CSS and JavaScript, you are losing visibility today.

Robots.txt is a tiny text file with a big job: it tells crawlers which parts of your site they may visit. One wrong line can remove a whole website from search. In audits I find a robots.txt problem on roughly one site in five.

1. Leaving “Disallow: /” from staging

Developers block crawlers while a site is being built, then forget to remove the rule at launch. In WordPress, also check Settings → Reading → “Discourage search engines” — it does the same thing through a noindex tag.

2. Blocking CSS, JavaScript or /wp-content/

Google renders pages like a browser. If it can’t load your stylesheets and scripts, it may see a broken layout and judge the page as poor on mobile. Never block /wp-content/themes/ or /wp-content/plugins/.

3. Using robots.txt to “remove” pages

Robots.txt stops crawling, not indexing. A blocked URL can still appear in Google with no description if other sites link to it. To remove a page, let it be crawled and add a noindex tag — I explain the difference in crawlability vs indexability.

4. Blocking AI crawlers by accident

Many security plugins and CDN settings now block bots such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. If you want to appear in AI answers, these crawlers need access. Decide on purpose instead of by default.

5. Wrong sitemap line

Add a full URL: Sitemap: https://example.com/sitemap.xml. A relative path or an old sitemap URL wastes the hint. See my XML sitemap checklist.

6. Case and slash errors

Rules are case-sensitive and match from the start of the path. “Disallow: /Blog” does not block /blog/, and “Disallow: /shop” also blocks /shopping-guide/.

7. Conflicting rules for different bots

When you add a group for a specific bot, that bot ignores the “*” group completely. Copy any rules it still needs.

How to check yours

• Open /robots.txt in a browser.
• In Google Search Console, open Settings → robots.txt to see the version Google fetched and any errors.
• Use the URL Inspection tool on your key pages to confirm “Crawl allowed: Yes”.

Robots.txt is the first thing I check in every technical & on-page SEO audit, because when it is wrong, nothing else matters.