Short answer: A robots.txt file is a plain text file at the root of your site, such as example.com/robots.txt, that tells crawlers which URLs they can access. Google says its main use is to avoid overloading your site with requests. It doesn’t keep a page out of Google: for that, use noindex or a password.
The rules a robots.txt file uses
| Rule | What it does | Example |
|---|---|---|
| user-agent | Names the crawler a group of rules applies to; every group starts with it | User-agent: Googlebot |
| disallow | A path that crawler shouldn’t crawl | Disallow: /nogooglebot/ |
| allow | A path it may crawl, even inside a disallowed folder | Allow: /wp-admin/admin-ajax.php |
| sitemap | The full address of a sitemap; optional | Sitemap: https://www.example.com/sitemap.xml |
| # | Starts a comment, which crawlers ignore | # Block only Googlebot |
Rules are case-sensitive, so Disallow: /file.asp covers /file.asp but not /FILE.asp. Every rule except sitemap accepts the * wildcard. User-agent: * covers every crawler except Google’s AdsBot crawlers, which must be named. Google doesn’t support other fields, such as crawl-delay.
From Google Search Central, “How to write and submit a robots.txt file” and “How Google interprets the robots.txt specification”, read November 2, 2026.
What robots.txt can’t do
A blocked page can still show up in Google. If other pages link to it, Google can index the URL without visiting the page, and the result appears without a description.
Not every search engine supports its rules, and no crawler is forced to obey them: Googlebot and other respectable crawlers do, but other crawlers might not.
Where the file goes and how Google reads it
The file must be named robots.txt, and a site can have only one. It goes at the root of the host, such as https://www.example.com/robots.txt, and it doesn’t cover subdomains like m.example.com.
It must be a UTF-8 text file. Write it in a text editor, not a word processor, which can add characters such as curly quotes that cause problems for crawlers.
Google reads up to 500 KiB and ignores anything after that. It usually caches the file for up to 24 hours, so a change can take a day to apply.
Robots.txt and AI crawlers
Google-Extended is the token that decides whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps. Google says it doesn’t affect inclusion or ranking in Google Search.
OpenAI gives site owners two robots.txt tags. OAI-SearchBot surfaces sites in ChatGPT search answers, and GPTBot crawls content that may be used to train its models. Each setting is independent, so a site can allow OAI-SearchBot and disallow GPTBot.
OpenAI says a change can take about 24 hours to reach its search results. Its ChatGPT-User agent visits pages when a person asks ChatGPT something, and robots.txt rules may not apply to it.
From Google’s list of common crawlers and OpenAI’s “Overview of OpenAI Crawlers”.
How to write your robots.txt
- Check whether your host manages the file for you: Wix and Blogger, for example, use search settings instead.
- Create a plain text file named robots.txt in a text editor.
- Start each group with a User-agent line that names the crawler, such as User-agent: Googlebot.
- Add Disallow and Allow lines with paths that start with a slash.
- Add a Sitemap line with the full address of your sitemap.
- Upload the file to the root of your site, so it opens at yoursite.com/robots.txt.
- Open it in a private browser window, then check the robots.txt report in Search Console.
Our SEO checklist covers the rest of what Google needs to reach and index a page.
What our own robots.txt says
Ours has four lines. It blocks the WordPress admin folder and allows one file inside it, admin-ajax.php. It also gives the address of our sitemap index, which lists one sitemap for posts and one for pages.
It names no AI crawler, so GPTBot, OAI-SearchBot and Google-Extended follow the same rules as other crawlers: only the admin folder is closed to them. We want AI assistants to be able to read our guides and cite them.
Before you upload robots.txt
- The file is named robots.txt and sits at the root.
- It is plain UTF-8 text, saved from a text editor.
- Each group starts with a User-agent line.
- No page you want on Google is disallowed.
- Pages you want out of Google use noindex, not robots.txt.
- The Sitemap line has the full address.
RememberRobots.txt manages crawling, not indexing. Keep it short, test it in a private window and in Search Console, and decide which AI crawlers you allow.
If you want us to review how Google crawls your site, see our technical SEO service.
Quick answers about robots.txt
Does every website need a robots.txt file?
No. Without one, crawlers may crawl the whole site, which is the default behavior Google describes. If the file answers with a 4xx error other than 429, Google treats the site as having no crawl restrictions.
Can robots.txt remove a page from Google?
No. If other pages link to a blocked page, Google can still show its URL in results, without a description. To keep a page out of Google, block indexing with noindex or protect the page with a password.
Where does the robots.txt file go?
At the root of the host it applies to, such as https://www.example.com/robots.txt. Its rules only cover that protocol, host and port, so a subdomain like m.example.com needs its own file.
How do I block AI crawlers like GPTBot?
Add a group for each crawler, such as User-agent: GPTBot followed by Disallow: /. For Gemini training and grounding, Google’s token is Google-Extended, which Google says doesn’t affect Google Search. OpenAI’s OAI-SearchBot is separate and decides whether you appear in ChatGPT search answers.
Does Google support crawl-delay?
No. Google reads the user-agent, allow, disallow and sitemap fields, and other fields such as crawl-delay aren’t supported. Google adds that 4xx codes other than 429 have no effect on how often it crawls.
Sources
Read on November 2, 2026. Google Search Central: Introduction to robots.txt, How to write and submit a robots.txt file, How Google interprets the robots.txt specification and Google’s common crawlers. OpenAI: Overview of OpenAI Crawlers. Our own case: our robots.txt, on November 2, 2026.