My log file is showing an entry with the following hostname: crawl-66-249-71-57.googlebot.com From what it is named, and please correct me if I am wrong, I am assuming this is googlebot crawling my site. However, the bot stopped at my default(main) page where there are links to other pages in the site. Why??? What happened?? Did I do something wrong?
I seriously doubt you did something wrong. With that said, Googlebots do not "Deep Crawl" on each visit.
I have only recently setup logging. And this is the first time I see an entry for googlebot. I know Google has indexed the site before I started logging. But since then I have made a lot of changes to make the site more SE friendly but it does not look like the bot is picking them up. My site, when listed in Google, there is only the url, no description, no text, no nothing....grrrrrrrrrrr...........
Step by step, that's how Goo does it. And you don't know when or why it's going for a deep crawl. At least I know nobody that exactly knows. I suppose you have to wait and invite the bot to come through again soon...
Googlebot hits my main page a couple of times a week. It only deep crawls twice a month. And when it does deep crawl, it doesn't happen all at one time ('crawl' isn't the right word visually) - Several Googlebot IPs hit it once every 5-10 minutes getting a new page each time. So, you are probably fine, wait a few weeks and see what it does after that.
Thanks for the replies guys. Cant say enough how much I have learnt from this board. It is the vacation site. Feel free to look around. It is just a simple site, welcome any suggestion. Is the board kinda flaky today, every maybe 20 mins I get a Page Not Found error and it will be okay after maybe 5 mins.
Ahhh, Kauai! My favorite place in the world!! Your site should be ok. Get some links, update your content frequently, maybe add a site map and you should be just fine.
Google won't do a deep crawl each time, even if it is on old and established site with a high PR. So, Googlebot stopping at the home page is fine (and normal). Nothing to worry about! I would recommend that you make some changes to your home page, add some text, move things around etc. so that the next time GoogleBot visits it sees an updated page. This should increase your chances of getting a deep crawl sooner, as GBot seems to love fresh content. Hope this helps!
Yes thats what I plan to do, to add some more text on the default page and later a site map. I still have to get with my clients to come up with a good text description. Oh and Kauai .... um..... not for me.... I still prefer the city
I have the exact opposite problem.. Google seems to deep crawl my site (almost) every day, but it wont touch my home page at all.. It usually spiders from 100-1000 internal pages daily, but the home page cache was last updated January 6, and December 24 before that.. It used to be spidered almost daily before.. It's a PR 6 by the way.. Anyone else seeing this???
Read articles like this one http://research.compaq.com/SRC/mercator/papers/www/paper.html to see how spiderbots work. He basically came to your homepage indexed it and copied and pasted all links he found into a queue. Another bot will go through that list of links and carry on the same process from there.
I wouldn't really call this a problem. At least Google is doing deep crawls. Most people complaint 'cos it doesn't. As far as your home page is concerned, at least it's being cached once every 10 days. Is it really a problem if your home page shows a 9 day old cache in Google?? Having said that, I must say that this is quite strange and I haven't heard anybody else report similar behaviour. Do most of your incoming links link to internal pages, as opposed to the home page? And, do you have a high PR page linking to an internal page??
It isn't exactly a problem.. But I get a lot of SE traffic directly to my home page, so I've used to experiment a bit with in the past when it was spidered and indexed/updated every 2-3 days. But now it's not possible to do that anymore since it's 14-16 days between the cache updates.. Most of my incoming links (15-20,000) are directed towards my homepage, but I've also got a few thousand links pointing to many different internal pages as well (both high and low PR links)..
G-bot deep crawls my e-commerce sites daily or every other day. I update the info automatically on my sites each night so it knows that it's going to get fresh stuff each time it visits.
A classic example of: One man's garbage is another man's treasure.... I'd love to take that problem from you...