link list; how much can a bot spider?

Discussion in 'Search Engine Optimization' started by nikao, Jul 31, 2006.

0
  1. #1
    for a website with over 8000 productpages I made a list with links to all the pages to make sure they get indexed.
    The page was crawled, but if I look at the cache, it didn't index the whole page. It stopped right before the end of the link list.

    Is there any indication on how much links can be crawled/indexed on 1 page?
     
    nikao, Jul 31, 2006 IP
  2. zubrag

    zubrag Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Google suggest having not more than 100 links on one page.
    See google.com/support/webmasters/bin/answer.py?answer=35769
     
    zubrag, Jul 31, 2006 IP
  3. nikao

    nikao Peon

    Messages:
    216
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #3
    yeah im aware of that, but that's just an indication of what google wants ;)

    it has indexed at least 7k links, so the 100 doesn't seem realistic ;)
     
    nikao, Jul 31, 2006 IP
  4. KC TAN

    KC TAN Well-Known Member

    Messages:
    4,792
    Likes Received:
    353
    Best Answers:
    0
    Trophy Points:
    155
    #4
    They can index more than several thousands of links, no problem... but your page will be treated as link farm to them..
     
    KC TAN, Jul 31, 2006 IP
  5. kajsng

    kajsng Peon

    Messages:
    74
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I'd suggest using the suggestion of 100 links/page. You can have one page with links to 80 sub-sitemaps with links to the 8000 product pages.

    If you really want to jam as many as you can on one page though, I've got an idea. You mentioned that it cut off near the end of the page - if you want the remaining pages indexed, just put the list in reverse order so it indexes the ones it hasn't indexed at all first.
     
    kajsng, Jul 31, 2006 IP