Robots.txt is usually a file that exists on the basis Listing of every Web page and can be used to instruct search engines like google and yahoo on which directories/information of the web site they're able to crawl and include inside their index. According to Rubright, a better tactic is https://news.rafeeg.ae