Hey there, This is a general question about Google and not about a specific site. What if somebody releases their site to Google by editing the robots.txt file, submits a google site map using webmaster tools, and Google does not crawl a website? How does somebody get Google to crawl their website if the sitemap doesnt work? I do suppose Google didnt stop crawling websites all of the sudden.. at least I would hope not Sincerely, Travis Walters
Well, as of my experience, more links you get for a specific page - more chances you'll be crawled. Content changes also matter. The sitemap values just give suggestions for the robots, they will not strictly follow them.. Could you describe your problem more precisely? You're not crawled at all? For how long? The entire site or just a page? all info, you can provide...
Hey Tsid, Thanks for the response. Well, it was released to Google 2 days ago. I thought Google would have at least crawled the home page by now, but my tracker doesnt show Google crawling. I guess I will just have to get more backlinks and list in directories and be patient. Sincerely, Travis Walters
Two days is not really much - i would begin to worry after a week of silence or so... Just a few days ago i've started a new website the same way on a newly-registered domain name. Got crawled after a day, then got a 2 days pause and then crawling continued by 5 pages a day or so. Another case - if you were building your site for some time right on place (i.e. in the root directory) google could find it by itself, start crawling, seen no content changes, etc... anyways, a week is a normal time to wait. No one likes it, but it is so =) Same for appearing in the index - for a new site.