Hey there, I have a rather unique problem. When I do a search for "www.lukebouman.com", the robots.txt file is showing as the first supplemental result. I do not really think I should have this in the disallow list in the robots.txt file... What should I do? Sincerely, Travis Walters
If you got google webmaster account, you can remove it from there and do chk your sitemaps if it's linking your robots or any link within your page that linking to your robots.txt
It is because you have this in the robots.txt file User-agent: * Disallow: Google reads this file first when it crawls a website and the way you have it set is telling google to not crawl your website. You need to edit the robots.txt file and allow google bot or just remove the entries in that file. Boulder
Hey there, No, that tells google to crawl the entire website. User-agent: * Disallow: / The above would be blocking everything from being spidered. I just want robots.txt out of my search results I sent in a request with webmaster tools Just waiting for a response now. Sincerely, Travis Walters