Introduction
The robots.txt file is an essential component in website management and technical SEO. This file instructs web crawlers to follow or avoid certain URLs in a website. Search engine bots, AI crawlers, and other automated agents may review a robots.txt file before visiting content.
While this file contains a simple set of instructions, special characters, unsupported characters and misunderstandings of directives can result in unpredictable crawling patterns. Understanding how the robots.txt file works under the official internet standard, RFC 9309, can help website owners exert better control over crawlers.
Robots.txt overview
A robots.txt file is a plain text file that is stored in a root folder of a website. It is used to provide instructions to crawlers.
For example, a website with a domain of https://example.com usually has robots.txt at the following address:
https://example.com/robots.txt
The file uses directives to provide instructions to crawlers. The most common directives include:
• User-Agent – specify a crawler that will receive the following instruction
• Disallow – ask crawlers to avoid URLs that match the following path
• Allow – permit accessing the following path
• Sitemap – provide the address of a sitemap
A simple example of a robots.txt file is given below:
User-agent:
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml
This example instructs compliant crawlers to avoid URLs that start with /private/ and permit visiting URLs that start with /public/. This file also specifies that a sitemap is located at the address https://example.com/sitemap.xml.
A note about security: the Robots Exclusion Protocol defines how web servers and web robots must behave. It does not define how confidential information must be protected nor does a Disallow directive prevent unauthorized visitors from visiting a URL.
Robots.txt and RFC 9309
RFC 9309 specifies the Robots Exclusion Protocol. This document describes how web crawlers must interpret robots.txt files and how website owners can write directives to influence crawlers. Search engine bots, such as Googlebot, use a robots.txt file to determine which URLs must not be crawled. However, this file is not an encryption method. Web servers and crawlers that do not follow the standard may ignore the file.
The standard also specifies how crawlers must interpret user-agent groups and path matching. Writing standard-compliant directives helps website owners obtain better control over crawlers.
Can I use ASCII art in robots.txt?
Since a robots.txt is a plain text file, it can include comments. Lines that start with a # character are treated as comments.
For example:
# Website crawling rules
# ———————-
# Please respect these preferences.
User-agent:
Disallow: /admin/
It is possible to add ASCII art to a robots.txt file. However, it is best to avoid adding unnecessary content to this file. Adding such content will not improve the SEO value nor will it help crawlers identify your site. Moreover, comment lines can interfere with directive lines. For example, the following code will cause unexpected results:
# Website crawling rules
User-agent:
Disallow: /admin/
In this example, a line with User-agent: is treated as a comment since this line starts with a # character. As a result, this directive will be ignored by a compliant crawler. As always, it is best to keep a robots.txt file simple. Avoid placing any unnecessary text, including comments, in the middle of directives.
How are formatting errors handled by crawlers?
A robots.txt parser knows which directives are supported by a crawler. If a directive is not supported, the parser skips the entire line.
For example:
User-agent:
Don’t crawl /secret/
The second line in this example is not a valid directive. Therefore, a compliant parser ignores this line. The correct code is given below:
User-agent:
Disallow: /secret/
It is essential to ensure that every directive is valid and will be interpreted by a crawler as intended. A small syntax error can cause a page to be crawled when it was supposed to be disallowed or vice versa.
Can I use an emoji in a user-agent?
A user-agent directive specifies a web robot that will receive the following instruction. This directive specifies a string that identifies a web crawler. The string can include emojis or other characters.
However, using such characters is not recommended. A robots.txt file is a text file and every character must be represented in a format that can be safely parsed by a crawler.
For example, the following code will be ignored by a compliant crawler:
User-agent: 🐔
Disallow: /
This example instructs web crawlers that have a user-agent of 🐔 to avoid crawling all website resources. However, this instruction will be ineffective since most crawlers do not support such a user-agent.
If a website owner wants to restrict a specific crawler, they must research the official name of this crawler and use this name in the robots.txt file. For example, the following code instructs Googlebot to avoid crawling all website resources:
User-agent: Googlebot
Disallow: /
A general rule of thumb is: always research the official name of a web crawler before specifying this name in the robots.txt file.
Robots.txt and internationalization
Website URLs can include characters from many alphabets, including Arabic, Hindi, Chinese, Japanese, and European languages. Moreover, URLs can also include emojis.
International characters in URLs are encoded using UTF-8 percent encoding. This encoding is critical for robots.txt.
For example, the following URL includes international characters:
https://example.com/caf%C3%A9/
The actual characters in this URL depend on how this URL was encoded. In general, most modern crawlers understand UTF-8 encoding, but there may be discrepancies in how different crawlers handle raw international characters or encoded URLs.
RFC 9309 specifies how web crawlers must interpret robots.txt directives and website URLs. Following these conventions can help avoid unexpected behavior.
Percent encoding in robots.txt
Percent encoding is a method of encoding bytes as characters. A percent symbol is always followed by two hexadecimal digits. This method is used to represent characters that are not allowed in URLs.
In general, you do not need to encode characters in robots.txt. However, there are some cases when percent encoding is required. For example, a website URL may include emojis or international characters.
When using such URLs in a robots.txt file, it is essential to use percent encoding. Otherwise, a crawler may not follow the specified instruction. For example, you may want to prevent crawlers from visiting URLs that have emojis in their paths. A robots.txt directive would look like this:
Disallow: /caf%C3%A9/
A compliant crawler will avoid visiting URLs that match this pattern. If you are unsure about the encoded value of a character, use an online encoder to obtain the required value.
When working with international URLs in robots.txt:
1. Always make sure to identify the exact URL path.
2. Encode this path using UTF-8 percent encoding.
3. Apply the changes to robots.txt and test this file with a robots.txt testing tool.
4. Ensure that crawlers that matter to you follow the new directives.
Never guess the encoded value of a character. A small discrepancy can cause unexpected results.
Robots.txt and Generative AI Crawlers
As generative AI continues to evolve, website owners are interested in learning how these systems parse content. Some companies utilize AI crawlers to build indexes, search databases, develop models, and provide other services. The ability to restrict a crawler depends on the company that operates it.
Robots.txt files can be used to instruct compatible AI crawlers, just as they can instruct other web crawlers. However, it is essential to remember that this file cannot restrict how information is processed after it has been collected.
Can I use robots.txt to stop AI training?
A robots.txt file is used to provide instructions to web crawlers.
For example, a website owner can request that a specific AI crawler avoids visiting a certain set of URLs. This restriction only affects crawling. It does not impact how the information is used afterwards.
A website owner can also request that an AI crawler avoids visiting an entire website:
User-agent: ExampleAIBot
Disallow: /
In this example, ExampleAIBot is a hypothetical identifier. This token should be replaced with the real identifier of an AI crawler. The Disallow: / directive requests that this crawler avoids visiting all website resources.
It is essential to ensure that the crawler actually respects robots.txt directives. This file is a guideline for compliant crawlers. It does not prevent malicious crawlers from collecting information. Moreover, a robots.txt file cannot prevent AI systems from using information that has already been collected.
Custom instructions are also ineffective. For example, the following code attempts to prevent a crawler from using website resources for AI training:
User-agent: ExampleAIBot
Disallow: /training/
This directive only works for robots that follow the Robots Exclusion Protocol. However, most AI crawlers do not follow this protocol. Therefore, this directive is ineffective.
If I want to restrict a specific AI crawler, what should I do?
Most AI services use several different crawlers to index websites. Some of these crawlers are similar to search engine bots, while others are designed to specifically train AI models.
To restrict a specific AI crawler:
1. Research the official name of a crawler that you want to restrict.
2. Review this crawler’s documentation to determine how robots.txt must be formatted.
3. Update your robots.txt file to include a Disallow directive for restricted paths or the entire site.
4. Review this crawler’s documentation to determine if there are additional controls for training data, search indexing and other purposes.
5. Review this company’s robots.txt documentation to understand how directives affect their crawlers.
6. Never assume that a Disallow directive will be followed by all crawlers operated by a company.
Every AI company utilizes different services and policies. Blocking one crawler does not always block all crawlers operated by the same company.
Example: I want to prevent my site from being crawled by a particular AI crawler
If a website owner wants to prevent a specific AI crawler from collecting information about their website, they can update their robots.txt file as follows:
User-agent: ExampleAIBot
Disallow: /
Where ExampleAIBot is replaced with the real name of an AI crawler. This update requests that ExampleAIBot avoids visiting all website resources.
Best practices for robots.txt
Robots.txt is a valuable tool for managing how web crawlers collect information about a website. A well-written robots.txt file:
1. Contains valid directives.
2. Keeps directives simple.
3. Uses the correct URL encoding.
4. Accurately specifies website resources.
5. Uses valid user-agent directives.
6. Avoids unnecessary comments.
7. Contains only necessary directives.
8. Prevents private information from being compromised.
9. Restricts generative AI crawlers appropriately.
10. Ensures that essential pages are not blocked.
Conclusion
Robots.txt is a small file that provides critical instructions to web crawlers. Following the standard described in RFC 9309, using valid directives, encoding international characters appropriately and correctly identifying crawlers can help avoid unexpected results. As generative AI crawlers continue to emerge, website owners must understand how to prevent these services from collecting information about their sites.
