I have a question about how crawlers usually work. Specifically, yesterday afternoon GoogleBot finally hit the main page of my site, but I haven't seen any crawling since then. Is that normal, or does it mean there's something wrong with my site such that the crawler can't find the links (or doesn't like them for some reason). I think they're fairly obvious. No weird scripting or dynamic pages or anything like that. Similar behavior from MSN and Yahoo. Yahoo has "slurped" the home page of my site 6 times in the last day (all from different IP's), but only hit one other page. Pretty much the same deal from MSN. I've submitted my sitemap to both google and yahoo, not that I was really hoping that would do much good... So what do you think? Is that normal, or do I have a problem? Anyone care to have a look at the HTML from my homepage? http://www.workfromhomespot.com Any wisdom much appreciated. -Gary
I assume this is a new site. That is pretty common behavior for spiders on a new site -- they will visit the home page and usually on that visit nor go beyond the home page, or not very far. Subsequently, you should expect return visits but they too may be only to the home page at first. Once your home page appears in the index, at some future point the spider will return to do a "deep crawl" where it follow links into your site from the home page. That's one of the reasons you should create a site map (I don't men the so-called Google SiteMap - that's something else) with text links to all your important pages and then make sure there is a text link on the home page somewhere linking to your site map. The spiders will hit that home page link and follow it to your site map - from there they will follow the links to the rest of your site. Example: see http://www.psychlinks.ca and note the link there to Site Map at http://www.psychlinks.ca/pages/sitemap.htm - then take a look at the format of that sitemap.
Thanks for the info. I do have a "real" site map on the site, linked at the top of the main page. Guess I just have to wait for google to find it. You mentioned I should look at the format of your sitemap at psychlinks.ca. Anything in particular I should be noting? Mine's here if you feel like having a quick look: http://www.workfromhomespot.com/sitemap.aspx Thanks again, Gary
Yours looks fine. I just gave you mine to look at as an example of one that was both spider friendly and user friendly (some visitors like to hit the sitemap instead of following other links or searching), in case you hadn't already created one. I stole the basic idea from apple.com