1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Yahoo just sent 188 spiders to my forum

Discussion in 'Yahoo' started by TheWebJunkie, Apr 27, 2005.

0
  1. #1
    Looks like yahoo has come to visit me 188 spiders were on my forums today :)

    [​IMG]
     
    TheWebJunkie, Apr 27, 2005 IP
  2. yfs1

    yfs1 User Title Not Found

    Messages:
    13,798
    Likes Received:
    922
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I was wondering what Yahoo has been smoking lately
     
    yfs1, Apr 27, 2005 IP
  3. just-4-teens

    just-4-teens Peon

    Messages:
    3,967
    Likes Received:
    168
    Best Answers:
    0
    Trophy Points:
    0
    #3
    maybe you should rub out you ip addy for security reasons?
     
    just-4-teens, Apr 27, 2005 IP
  4. Padawan

    Padawan Peon

    Messages:
    1,085
    Likes Received:
    70
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Perhaps you should get them to sign-up and keep you company :)
     
    Padawan, Apr 27, 2005 IP
  5. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #5
    If you're worrying about Yahoo! Slurp or MSNbot being overly aggressive in spidering your site, both will obey the crawler-delay instruction -- add this line to your robots.txt file:

    user-agent: *
crawl-delay: 120
    Code (markup):
    where 120 indicates the number of seconds between page retrievals... set this to whatever you wish (120 is probably a lot longer than necessary - maybe 10 - 30 seconds is more reasonable).
     
    minstrel, Apr 27, 2005 IP
  6. TheWebJunkie

    TheWebJunkie Banned

    Messages:
    630
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Thanks for the great tip :)
     
    TheWebJunkie, Apr 27, 2005 IP
  7. av1

    av1 Active Member

    Messages:
    168
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    60
    #7
    another great tip: you could send em to my site if you get hit too hard by em ;)
     
    av1, Apr 27, 2005 IP
  8. Stin

    Stin Guest

    Messages:
    264
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Yeah when Yahoo last deep crawled my site i got hit by atleast 130 different IPs. It was crazy to watch, because my site keeps track of guests as well. And it was just getting bigger and bigger as I was surfin around. Good times, wish yahoo would do it again.
     
    Stin, Apr 27, 2005 IP
  9. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #9
    Why? Did it do you any good?

    There were a lot of weeks a few months back where Yahoo was the busiest spider on the block by far for my sites -- I didn't see a lot of evidence of it doing anything very useful with all that spidering, though.
     
    minstrel, Apr 27, 2005 IP
  10. Stin

    Stin Guest

    Messages:
    264
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Cant remember if it did any good, i quickly moved to a new domain, currently they only have my main page indexed.
     
    Stin, Apr 27, 2005 IP