Robots.txt Can be a file that exists on the root directory of each Internet site and can be used to instruct search engines like google and yahoo on which directories/files of the website they're able to crawl and include inside their index. Based on Rubright, a far better approach is https://news.rafeeg.ae