1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

over 500 spiders from Yahoo on a daily basis

Discussion in 'Yahoo' started by Devilfish, Mar 28, 2008.

0
Page 1 of 3
  1. #1
    I have a forum and I get 500+ (sometimes 800+) Yahoo spiders daily. Why is there so many? I only get 1 or 2 Google spiders.

    I guess that's the difference is the algorithms and the technology they use.
     
    Devilfish, Mar 28, 2008 IP
  2. SticKer

    SticKer Well-Known Member

    Messages:
    2,393
    Likes Received:
    78
    Best Answers:
    0
    Trophy Points:
    115
    #2
    i don't know but for the past 3-4 days yahoo bots are acting weird. I see 50-60 bots all over my site at a given time and then they are all gone again after sometime they come back. Today it seems to be normal.
     
    SticKer, Mar 28, 2008 IP
  3. jpl691

    jpl691 Well-Known Member

    Messages:
    348
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    113
    #3
    yeh the google bot i think lives at my forum
     
    jpl691, Mar 28, 2008 IP
  4. leo7068

    leo7068 Active Member

    Messages:
    28
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    66
    #4
    you can limit the spiders with robots.txt right?
     
    leo7068, Mar 28, 2008 IP
  5. Devilfish

    Devilfish Active Member

    Messages:
    396
    Likes Received:
    6
    Best Answers:
    1
    Trophy Points:
    70
    #5
    I know you can limit the bot visits but my forum is new(ish) so I need all the visibility I can get lol.

    Just wondered, that's all. :)
     
    Devilfish, Mar 28, 2008 IP
  6. Gnome

    Gnome Active Member

    Messages:
    236
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    58
    #6
    Yeah i noticed lots of yahoo bots around.
    And google lives at my forum as well :D
     
    Gnome, Mar 28, 2008 IP
  7. iceman2000

    iceman2000 Peon

    Messages:
    129
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I am new to this.

    How do you determine that bots are on your site and how do you limit their visits?

    I am assuming that everytime they visit, they chew up some bandwidth.
     
    iceman2000, Mar 28, 2008 IP
  8. snappingpig

    snappingpig Banned

    Messages:
    331
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #8
    I get the same thing, Yahoo just seems crappy for some reason because I'm never on there. Google is the best though and they can get the job done every day without 100s of spiders.
     
    snappingpig, Mar 28, 2008 IP
  9. soccerfriend

    soccerfriend Well-Known Member

    Messages:
    1,060
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    128
    #9
    same here I had 1000 ++ yahoo spiders.. maybe yahoo improving their search engine :)
     
    soccerfriend, Mar 28, 2008 IP
  10. Neticule

    Neticule Peon

    Messages:
    491
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #10
    If its a forum would it say there are 1000+ guests viewing? lol
     
    Neticule, Mar 28, 2008 IP
  11. cheaptraffic

    cheaptraffic Active Member

    Messages:
    398
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    60
    #11
    If too many bots visits your site, would it make your site slower?
     
    cheaptraffic, Mar 28, 2008 IP
  12. high_flying

    high_flying Banned

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    No not really. Especially if the images and html files are small
     
    high_flying, Mar 29, 2008 IP
  13. obay

    obay Notable Member

    Messages:
    2,458
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    205
    #13
    obay, Mar 29, 2008 IP
  14. rsfrs

    rsfrs Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    I think better u control from robot.txt
     
    rsfrs, Mar 29, 2008 IP
  15. sachin410

    sachin410 Illustrious Member

    Messages:
    6,422
    Likes Received:
    573
    Best Answers:
    0
    Trophy Points:
    410
    #15
    same with my forum.

    Yahoo bots are on steroids.

    can someone please explain how?
     
    sachin410, Mar 29, 2008 IP
  16. djixas

    djixas Well-Known Member

    Messages:
    749
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    110
    #16
    I got hundreds of connections at same time from Google spiders which crashed my old server once (when Yahoo did the same).
     
    djixas, Mar 29, 2008 IP
  17. FraserJilani

    FraserJilani Active Member

    Messages:
    174
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    58
    #17
    Getting 800 - 1000 Yahoo per day on my forum. Sometimes 50 + at one moment, yet only had 6 Google spiders visit in the past 24hours.

    Wierd.
     
    FraserJilani, Mar 29, 2008 IP
  18. joeschmo12

    joeschmo12 Peon

    Messages:
    107
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    yeah yahoo spiders.. aka yahoo slurp spiders... they just never seem to leave your forums.. whereas google sends a few spiders a day and can get the same amount of info. Shouldnt that send a message to yahoo about fixing their slurps?
     
    joeschmo12, Mar 29, 2008 IP
  19. sydneyaus

    sydneyaus Active Member

    Messages:
    1,110
    Likes Received:
    44
    Best Answers:
    0
    Trophy Points:
    88
    #19
    The past few days Yahoo has crawled my site daily 70-80 times 1 page at the time. Google just comes by 5 times a day and takes 25-40 pages at a time. strange
     
    sydneyaus, Mar 29, 2008 IP
  20. Jim4767

    Jim4767 Prominent Member

    Messages:
    4,738
    Likes Received:
    766
    Best Answers:
    10
    Trophy Points:
    305
    #20
    I'm using StatCounter. Can anyone who knows StatCounter point me to the portion of StatCounter where I can check on the frequency of the spiders' crawling activity? Thanks.
     
    Jim4767, Mar 29, 2008 IP
Page 1 of 3