I did something amazingly dumb. I changed hosts recently, as my old host has been giving great problems. In the process, I accidentally changed my robots.txt to User-agent: * Disallow: * Doh! Now, Google has (obviously) started to remove my listings! I've fixed robots.txt as it should be. How can I get Google to re-read robots.txt quickly? The website is www.landau.ws Thank you!
If you buy some traffic (yip its worthless for visitors) but i assure your site will be super indexed by all SEs. "Traffic section DP" I did it with a domain that was barely getting indexed and within 2 weeks the site had over 1000 indexed pages. You might also want to get a few backlinks to help your site as building backlinks never hurts but only helps. thx malcolm
I fail to see how buying traffic is going to make any search engine index your pages quicker. You would be better off submitting an up to date sitemap to Google and if you have already submitted one, then re-submit and hopefully that will get Google to download your new robots.txt file
Hello... All i can tell you is that the site wasnt getting indexed very well and after i did the traffic thing it started getting spiders coming from everywhere which helped the indexing! (from 200 indexed to 800 within 2 weeks) I have nothing to gain from saying this as i dont own any "traffic site" but can say that the results where well worth the $20 bucks i spent. Maybe the traffic was mostly bots? dont know dont care it worked As for submitting a sitemap.... thats a given/must thx malcolm
Unfortunately, Google says that it can't download my sitemap because of the robots.txt file. The problem is that Google hasn't looked at my restored robots.txt file -- that's what I really needed it to do as the very first step!
Google ordinarily scans the robots.txt file before each crawl session, so its probably found your changes already. Keep in mind that the reports in the Webmaster Tools console are often 1-2 weeks out of date. I'm sure if you request a new crawl of your sitemap through the console, it will update properly within a few hours.
Thanks for the reassurance. I've just checked, and you're right... Google has indeed re-indexed robots.txt, and yes, the Webmaster Tools area says that it hasn't! I'm curious as to how long Google will take to catch up.