I have three new websites, all hosted in the same account, that I put on the web like 2 weeks ago. When I check the Awstats in my cpanel, it still says that Google Bot has not visited them, but others like Yahoo Slurp has. Why is this? I have link exchanges for these and done what I needed to do to get Google to crawl these sites. What could it be? It's really bothering me.
Ok I did that just recently. This is what I'm getting from google... "Network unreachable: robots.txt unreachable We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. " Which is where I also think the problem is coming in when referring to why Google bots arent able to index my site... Also, I used xml-sitemaps.com to create the sitemap.
Generally for any website once you are getting back links towards to your website, your website getting indexed in every search engines automatically. still you need to submit a sitemap, if you are not getting results for such a long time.
http://www.google.com/addurl/?continue=/addurl http://www.google.com/webmasters/ https://siteexplorer.search.yahoo.com/ put in your site robots.txt
You want to add a sitemap.txt not a robots.txt. Adding a robots.txt will not speed up the process and if you get it wrong, may actually make the problem worse. I set up a brand new site www.village-hall.org.uk about 10 days ago, entered the sitemap.txt into Google.com/webmasters and hey presto the Google bot had indexed the page in about 3 days.
There can be a number of reason why this is happening. But the closest ones are: 1. Your robots.txt is not configured properly 2. Your host is blocking the Google bot You can pin-point the problem if you do a HTTP header check under Google-bot from a Free HTTP Header Checking Tool.
I have already submitted a sitemap but Google Webmaster Tools is giving me the error saying "Network unreachable: robots.txt unreachable" when it tries to access it. I have some quality backlinks. I have submitted my URL to http://www.google.com/addurl/?continue=/addurl a few weeks ago. Still no luck. Another reason why I know something is wrong is because Google Bot will show up in my log of recent visitors but it never appears in my Awstats. It makes it seem like it has been trying to access the site but it isn't allowed to.
the sitemap.txt file should contain a straightforward list of all your page urls (including http://) for example: http://www.yoursite.com http://www.yoursite.com/index.htm http://www.yoursite.com/page2.htm http://www.yoursite.com/page3.htm and so on...