robots.txt

Robots.txt is a file that website owners use to guide web crawlers—automated programs that index websites for search engines. This file tells these crawlers which parts of a site they can or cannot access. For example, a website might want to restrict crawlers from visiting certain pages to protect sensitive information or to save server resources. Essentially, it helps manage how search engines interact with the site, influencing which pages appear in search results while respecting the site's privacy and structure.