Regardless that the robots.txt file was invented to tell engines like google what internet pages to not crawl, the robots.txt file can be used to stage search engines like google and yahoo on the XML sitemap. That is supported by Google, Bing, Yahoo and Talk to. Now, Google can’t understand https://rankingseotoolz.com/robots-txt-generator