"Last-Modified Header/If-Modified-Since" header and BOTS

Anyone have any experience with how the google bot (or any bot for that matter) interacts with these headers? If it interacts with these headers (or are they only for browsers)? I have a series of pages that I'm trying to get indexed but I'm having trouble..
The pages are:
3 pages deep from the homepage.
No more than 200 links per page
Titles and descriptions are well formed

Just seems that the bot gets tripped up on the second tier of pages. Wondering if the appropriate use of these headers will allow the bot to move on. Currently, I do not use them at all.

Any help would be great.

 

I think google has decided that your series of pages with 200 links is not worth indexing. Nothing to do with the headers.

Try get some deep external links or make some pages with less links.

 

Google suggests 100 links or less per page in their Webmaster Guidelines.

These two headers will help the spiders visit you more efficiently. When Google visits your site they basically ask if you've changed the page since their last visit (If-Modified-Since). If your script/server is setup correctly, it should respond with either a 304 (Not Modified) or it will proceed to serve the page.

Simply saying "304" is much easier on your system than spitting out the entire page and it's quicker for Google than having to check your page against it's current data. They'll have more time to spend on your pages that DID change since their last visit, but it won't be enough to convince them to spider a particular page or not.

 

Had any experience with Google then getting a 200 OK on your page, and yet still keeping the old page in the cache for days and days? I have a page, put up on the 19th, spidered on the 19th, 23rd, 25th and 26th, shows great in Yahoo! and MSN, but is still showing the old cache on Google.

Something is not working here the same way it was meant to in Matt Cutts blogs.

 

The cached page doesn't update for a few weeks after the page is spidered.