1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

"Last-Modified Header/If-Modified-Since" header and BOTS

Discussion in 'Google Sitemaps' started by cffoodie, Sep 21, 2006.

  1. #1
    Anyone have any experience with how the google bot (or any bot for that matter) interacts with these headers? If it interacts with these headers (or are they only for browsers)? I have a series of pages that I'm trying to get indexed but I'm having trouble..
    The pages are:
    3 pages deep from the homepage.
    No more than 200 links per page
    Titles and descriptions are well formed

    Just seems that the bot gets tripped up on the second tier of pages. Wondering if the appropriate use of these headers will allow the bot to move on. Currently, I do not use them at all.
    SEMrush
    Any help would be great.
     
    cffoodie, Sep 21, 2006 IP
    SEMrush
  2. mad4

    mad4 Peon

    Messages:
    6,987
    Likes Received:
    493
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I think google has decided that your series of pages with 200 links is not worth indexing. Nothing to do with the headers.

    Try get some deep external links or make some pages with less links.
     
    mad4, Sep 21, 2006 IP
  3. MaxPowers

    MaxPowers Peon

    Messages:
    261
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Google suggests 100 links or less per page in their Webmaster Guidelines.

    These two headers will help the spiders visit you more efficiently. When Google visits your site they basically ask if you've changed the page since their last visit (If-Modified-Since). If your script/server is setup correctly, it should respond with either a 304 (Not Modified) or it will proceed to serve the page.

    Simply saying "304" is much easier on your system than spitting out the entire page and it's quicker for Google than having to check your page against it's current data. They'll have more time to spend on your pages that DID change since their last visit, but it won't be enough to convince them to spider a particular page or not.
     
    MaxPowers, Sep 22, 2006 IP
  4. JetteroHeller

    JetteroHeller Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Had any experience with Google then getting a 200 OK on your page, and yet still keeping the old page in the cache for days and days? I have a page, put up on the 19th, spidered on the 19th, 23rd, 25th and 26th, shows great in Yahoo! and MSN, but is still showing the old cache on Google.

    Something is not working here the same way it was meant to in Matt Cutts blogs.
     
    JetteroHeller, Sep 27, 2006 IP
  5. mad4

    mad4 Peon

    Messages:
    6,987
    Likes Received:
    493
    Best Answers:
    0
    Trophy Points:
    0
    #5
    The cached page doesn't update for a few weeks after the page is spidered.
     
    mad4, Sep 28, 2006 IP