Get a free consultation
Website Design

Robots.txt in 2026: What Google Supports, What It Ignores, and AI Crawlers

By Sigmato Editorial TeamUpdated 15 Nov 20266 min read

Weathered blue wooden doors with padlocks, showcasing rustic charm and security.

Key takeaways

  • Google supports four robots.txt fields: user-agent, allow, disallow and sitemap. Fields such as crawl-delay are not supported.
  • Robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a snippet. Use noindex to keep a page out of search, and do not block that page.
  • Google-Extended is a robots.txt token that controls use of your content for Gemini training and grounding. It does not affect Search inclusion or ranking.
  • Google says it intends to support the shared AI training preference signal in early 2027. It also says Google Search does not use llms.txt.
  • A broken robots.txt can be costly: a server error stops crawling for up to 12 hours, and a forgotten Disallow: / from staging can hide a whole site.
Weathered blue wooden doors with padlocks, showcasing rustic charm and security.

The rules Google follows

Robots.txt looks simple, which is why it is so often misused. Here is what Google’s documentation says it supports and how it behaves.

Topic What Google does
Supported fields user-agent, allow, disallow and sitemap. Other fields, including crawl-delay, are not supported.
Where the file lives At the top level of the host. It applies only to that host, protocol and port. Crawlers do not look for robots.txt files in subfolders.
Size limit 500 KiB. Anything after that is ignored.
4xx response (except 429) Treated as if there is no robots.txt, so there are no crawl restrictions.
5xx response Google stops crawling the site for the first 12 hours while it retries, then for up to 30 days uses the last good cached copy. If there is none, it assumes no restrictions.
Caching Generally up to 24 hours, sometimes longer.
Case Field names and user-agent values are case-insensitive. Paths in allow and disallow are case-sensitive.
Wildcards * matches any run of characters and $ marks the end of the URL.

Google also clarified on 7 October 2026 that fields not listed in its documentation are not supported, and on 23 October 2026 added a recommendation to percent-encode rule paths. Raw UTF-8 and percent-encoded paths are both canonicalised by Google, but percent-encoding makes the file easier to process with other tools.

What robots.txt cannot do

Blocking a URL in robots.txt stops Google fetching the content, but “Google can’t index the content of disallowed pages, but it may still index the URL” and show it without a snippet. If you want a page removed from search, you need a noindex rule, and the page must stay crawlable so Google can see that rule. The classic mistake is to add Disallow for a page and a noindex tag on the same page, which means Google never reads the tag.

A safe starting template

For a typical WordPress site, something like this is a sensible baseline. Adjust the paths to your own site.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

Notice what is missing on purpose: it does not block /wp-content/, /wp-includes/ or your CSS and JavaScript files. Google needs those to render your pages.

AI crawlers and Google-Extended

This is the part of robots.txt that changed most in the last two years. Google separates the crawler that fetches pages from the token that says how the content may be used.

  • Googlebot crawls for Google Search, including Discover. Blocking it removes you from Search.
  • Google-Extended is a robots.txt token only. Google says publishers can use it to control whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI. It “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”
  • Google-InspectionTool affects testing tools such as the Rich Result Test and URL Inspection, and has no effect on Search.

If you decide you do not want your content used for Gemini training, the rule is:

User-agent: Google-Extended
Disallow: /

Whether to do that is a business decision, not an SEO one. It does not change how Google Search treats you. Other companies run their own crawlers with their own tokens, so check each one’s documentation if you want to manage them too.

What Google plans next, and what it does not use

  • A shared AI training preference signal. Google’s crawler documentation says it intends to support the AIPREF shared machine-readable training preference signal, using the token __usage_ai-training, in early 2027. A note about this was added to the changelog on 30 October 2026. Treat it as a plan, not something to implement today.
  • llms.txt. A note added on 15 June 2026 to Google’s AI optimization guide says Google Search does not use llms.txt files. Creating one for other services neither helps nor harms your Search visibility.
  • No robots.txt needed to be indexed. In January 2026 a rumour claimed a robots.txt file had become mandatory. Google’s response was that nothing had changed: with no robots.txt, crawling is assumed to be allowed.

Slowing Google down without blocking it

Because crawl-delay is not supported, the way to ask Google to ease off is to respond with 503 or 429 status codes, and Google’s guide to reducing crawl rate now includes details and examples of the Retry-After header (added 6 October 2026). Use this sparingly and temporarily. Returning errors for long periods can cause pages to drop out of the index.

The mistakes we see most often

Mistake What happens Fix
Staging Disallow: / left on the live site Crawling stops and pages disappear over time Check robots.txt in your launch checklist
Blocking CSS and JavaScript Google cannot render the page properly Allow the files pages need
Using robots.txt to hide a page URL can still appear without a snippet Use noindex and leave the page crawlable
Adding noindex inside robots.txt Not a supported field Put noindex in a meta tag or header
Using crawl-delay Google ignores it Use proper server capacity or temporary 503/429 responses
Wrong location or a file over 500 KiB Rules ignored or truncated Put it at the root and keep it short
Wrong letter case in paths Rule does not match Match the exact case of real URLs
Robots.txt returning a server error Crawling pauses for up to 12 hours Monitor the file and keep it fast and stable

How to test and monitor

  1. Open https://yoursite.com/robots.txt in a browser and confirm it loads with a 200 status.
  2. Use the robots.txt report in Search Console to see which version Google fetched and any errors.
  3. Test important URLs with URL Inspection to check they are not blocked.
  4. Put a robots.txt check in your deployment checklist so a staging rule never reaches production.
  5. Keep the file in version control and review changes like any other code.

Robots.txt is one small piece of technical SEO, alongside sitemaps, canonicals and redirects. Our guide to website redesign SEO shows where it fits in a launch plan. If you would like us to review your crawl setup, our maintenance and security team does this regularly, and you can get in touch.

Frequently asked questions

Does Google support crawl-delay in robots.txt?

No. Google supports only user-agent, allow, disallow and sitemap. Fields such as crawl-delay are not supported.

Can I use robots.txt to keep a page out of Google?

Not reliably. Blocking a URL stops Google fetching its content, but the URL may still be indexed and shown without a snippet. Use noindex and keep the page crawlable so Google can see it.

Does blocking Google-Extended hurt my search rankings?

No. Google says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. It controls use of content for Gemini training and grounding.

Does Google use llms.txt?

No. A note added on 15 June 2026 to Google’s AI optimization guide says Google Search does not use llms.txt files.

What happens if my robots.txt returns a server error?

Google stops crawling the site for the first 12 hours while it retries, then for up to 30 days uses the last good cached copy. If there is none, it assumes no restrictions.

Do I need a robots.txt file at all?

No. If there is none, crawling is assumed to be allowed. You still need one if you want to block specific paths or point to your sitemap.

Sources and further reading

Sigmato Editorial Team

Written by

Sigmato Editorial Team

The Sigmato Editorial Team is made up of software engineers, UI/UX designers and AI specialists at Sigmato Solutions Pvt Ltd in Bengaluru, India and the UK. We write practical guides on web, mobile, cloud and AI development, drawing on the products we design and build for startups and enterprises worldwide.

Last updated 15 November 2026 · Talk to our team