Looksmart and Inktomi crawlers

Discussion in 'All Other Search Engines' started by Roze, May 17, 2005.

  1. #1
    Looksmart and Inktomi are slowing my site to a crawl today. I just added a crawl-delay of 30 seconds to my robots.txt but I'm not sure they're obeying it... do they?

    My visitors can't even get to my site today, should I just block them??

    On a related note, does Googlebot 2.1 obey crawl-delay, I can't find a solid answer.
     
    Roze, May 17, 2005 IP
  2. newbie100

    newbie100 Peon

    Messages:
    17
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Most bots only read the robots.txt once every 24 hours, therefore any changes made today will unlikely alter their indexing.
     
    newbie100, May 17, 2005 IP
  3. Roze

    Roze Guest

    Messages:
    403
    Likes Received:
    29
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Ok, good to know, thank you!
     
    Roze, May 17, 2005 IP
  4. M.c

    M.c Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hello Rose***

    Please Can You Write To Me SO I can Knw or Think Of A Wai To Assist You Asap** My Email is maccollarme@yahoo.com


    Thnx For Ur Understandn. (+2348062409580)
     
    M.c, Apr 24, 2012 IP
  5. ashleyjohn2347

    ashleyjohn2347 Peon

    Messages:
    269
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I am not familiar with this thing right now. But want to know about this in future may be it would prove beneficial for me.
     
    ashleyjohn2347, Apr 26, 2012 IP
  6. Abbey14

    Abbey14 Peon

    Messages:
    164
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    It sounds new to me. Though, I better stick on the main search engines for me to market more my site.
     
    Abbey14, Apr 28, 2012 IP
  7. roxanne77

    roxanne77 Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Looksmart and Inktomi crawlers
     
    roxanne77, May 11, 2012 IP