Hey guys, I'm wondering if there is anything you can do to get google to deep crawl your page. I've been adding SSI scripts that give me daily fresh content on my pages and have been trying to link the deeper pages of my site, but googlebot still won't go near my deep pages nearly as often as I would like him to. What cologne is googlebot most attracted to. But seriously, does anyone have any good tips for getting the real deep pages of your site crawled regularly? Thanks, Barrow
Bot likes cookies. I put cookies.php, chocolate-cookie.html, chocolate-hobnob.asp etc. all over my site. Then watch him run around like a mad puppy. Seriously? Links. That's it. Think you have enough? Think again. Get more.
Ahah! Leave cookies out for Mr. Googlebot! I think it's time for oreo.shtml and some incoming links from gotmilk.com.
It's good idea to provide them with an easy way to find your pages. It always helps to have a sitemap or an archive in the case of forum sites.
Links are defenitly number one. Every time I go on a linking rampage and get a few more links to my site I can be sure to get a deep crawl within a day or two. If fact, over the weekend I hit a bunch of new directories and now Gbot has been at mysite for nearly three days! I'm not sure if that's good or bad, but every little bit helps.
definetly links.. after joining this forum.. i got over 200 hits from googlebots within a couple days. but the msn bot has been going crazy the past month or so
The above is all good advice - something additional for thought. Don't just get links, but link deep. That is, link to key internal pages (even if they're at the 'bottom' of your nav tree - though not orphans of course) from outside your site. You probably figured this, but thought I'd offer it anyway. Cheers, JL
Google or any other bot never starts indexing your home page and then crawls your deeper pages. Google does shallow random crawls across various sections of your website in order to find the must lucrative section (with content). In order to get it to reach those pages - link internal pages so that they are never deeper then few clicks may be 3-4 - link your home page around the web but also link to the inner deeper pages as these will become the doorways for future crawling - internal link building should be very strong since if google cannot find a page, it certainly can never be crawled and indexed
This may be an old forum thread, but I found this information very useful - especially linking to specific pages, not your base site page. Thanks!