Definición:
Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. The authors of the web pages can use robots.txt so that the robots that participate in the tracking cooperate and do not have access to the entire site or parts of a website that they want to keep private.
Functionality of the robots.txt file
Robots.txt allows webmasters to tell search engines which parts of the site should be excluded from crawling and has great relevance to technical SEO in terms of site indexing. Although it is not a method of protecting sensitive content, as it does not prevent direct access to URLs, it is an effective way to manage crawling and indexing of unwanted content. Search engines generally adhere to the guidelines set out in it, although they are not obliged to do so.
Importance and limitations of robots.txt
When assessing the importance of this file it is important to consider the following:
- Crawl control: The robots.txt helps direct traffic from search bots, optimizing server bandwidth usage and improving crawl efficiency.
- It is not a security tool: It should not be used to hide sensitive information, as it does not prevent direct access to URLs. For content protection, appropriate authentication and authorization methods should be used.
- Voluntary guidelines: Although major search engines, such as Google and Bing, respect robots.txt instructions, there is no guarantee that all bots do, especially malicious bots.
Location of robots.txt
The location of robots.txt is very important. It must be in the main directory because otherwise search engines will not be able to find it. If the file is not found in this location, search engines will assume it does not exist and will proceed to crawl and index all site content. Examples of robots.txt usage
Specific Folder Tracking Lock:
User-agent: *
Disallow: /privado/
Disallow: /configuracion/Enable Site Wide Tracking:
User-agent: *
Disallow:Locking a Specific File
User-agent: *
Disallow: /archivo-secreto.html