googlebot/yahooslurp

Discussion in 'Search Engine Optimization' started by tony84, Jan 14, 2007.

  1. #1
    They have both been on my site for about 2 days non stop (or atleast everytime i check theyre there.

    Not sure if this is due to my site just being uploaded but there cant be any more than about 500 pages (its a forum with 170 members and about 600 posts 200 topics)
    They shouldnt spend that long on there should they?

    As it happens theyve not used up anymore than about 30mb bandwidth each so its not the end of the world but if this goes onto the end of the month its gonna be a fair amount of bandwidth they use up.

    Is this normal? If not how do i stop it? Obviously do want them to visit ive just never had a site where a bot stays for 2 days - let aloe 2 of them.
     
    tony84, Jan 14, 2007 IP
  2. classifieds

    classifieds Sopchoppy Flash

    Messages:
    825
    Likes Received:
    51
    Best Answers:
    0
    Trophy Points:
    150
    #2
    You can slow down google by adding the site to webmaster tools - https://www.google.com/webmasters/tools/siteoverview . Once its in, go to the Crawl Rate page and change the rate from Normal to Slow. I don't think there's anything you do with Yahoo.

    Personally I wouldn't worry about it and would be happy. If they're not crawling then you're not getting pages in the index.

    -jay
     
    classifieds, Jan 14, 2007 IP
  3. tony84

    tony84 Well-Known Member

    Messages:
    1,864
    Likes Received:
    29
    Best Answers:
    0
    Trophy Points:
    140
    #3
    google has finally goe after 3-4 days yahoo is still there, any suggestions?

    i dont mind them coming and going but 50mb of baswidth in 1 go is just rediculous. im hoping it was a 1 off due to me changing everything
     
    tony84, Jan 15, 2007 IP
  4. rizal

    rizal Peon

    Messages:
    85
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I agree about that.. They crawling that why our page got index
     
    rizal, Jan 15, 2007 IP
  5. Angelus

    Angelus Well-Known Member

    Messages:
    1,622
    Likes Received:
    52
    Best Answers:
    0
    Trophy Points:
    138
    #5
    Google bot came 215 times in 14 days to my site and I'm very happy about it. I don't know why are you woried?

    p.l.u.r.
     
    Angelus, Jan 15, 2007 IP
  6. tony84

    tony84 Well-Known Member

    Messages:
    1,864
    Likes Received:
    29
    Best Answers:
    0
    Trophy Points:
    140
    #6
    ive never had a site where they have been on my site pretty much continuously for 4 days. Not only is there 1 but 2 of them.

    Its as though theyre stuck on my site so, im not complaining at the minute but if it carries on then im gona have a lot of bandwidth usage.
     
    tony84, Jan 15, 2007 IP
  7. mad4

    mad4 Peon

    Messages:
    6,986
    Likes Received:
    493
    Best Answers:
    0
    Trophy Points:
    0
    #7
    If you want to rank well you need to let them spider as much as they want. They won't be getting stuck on your site unless you have session id's and they are thinking you have thousands of pages.

    Don't restrict the spidering as you won't be able to get your site indexed as quickly.

    Lots of spidering is a very good sign google trusts your site.
     
    mad4, Jan 15, 2007 IP
  8. oddcomments

    oddcomments Peon

    Messages:
    77
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #8
    That is most likely the case.

    In order to keep their index fresh... the more a site is updated, the more the spiders will crawl it. Once it sees that things are not changing as fast, it should slow down it's rate of return.
     
    oddcomments, Jan 15, 2007 IP
  9. oseymour

    oseymour Well-Known Member

    Messages:
    3,960
    Likes Received:
    92
    Best Answers:
    0
    Trophy Points:
    135
    #9
    oseymour, Jan 15, 2007 IP
  10. Angelus

    Angelus Well-Known Member

    Messages:
    1,622
    Likes Received:
    52
    Best Answers:
    0
    Trophy Points:
    138
    #10
    How much bandwtih was it? Couple of MB isn't that much...

    p.l.u.r.
     
    Angelus, Jan 15, 2007 IP
  11. manwanis

    manwanis Peon

    Messages:
    44
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    How do you judge that yahoo is on your site. I understand you can use sitemap to check google visit.
     
    manwanis, Jan 16, 2007 IP
  12. maldives

    maldives Prominent Member

    Messages:
    7,187
    Likes Received:
    902
    Best Answers:
    0
    Trophy Points:
    310
    #12
    Frequent visits = better crawling = better indexing of pages!

    There is nothing to worry about unless you are using free webhosting with limited bandwith. For serious bigger websites it is a good sign. ;)
     
    maldives, Jan 16, 2007 IP
  13. tony84

    tony84 Well-Known Member

    Messages:
    1,864
    Likes Received:
    29
    Best Answers:
    0
    Trophy Points:
    140
    #13
    im not on free hosting but my site is quite small, 100 reigstered users, bandwidth isnt too much but its about 1/10th if not more of my monthly limit, which is fine for the time being but if it gets more then im gonna need to increase it
     
    tony84, Jan 20, 2007 IP
  14. oseymour

    oseymour Well-Known Member

    Messages:
    3,960
    Likes Received:
    92
    Best Answers:
    0
    Trophy Points:
    135
    #14
    In my clients case I had to do something because he had about two dozen spiders eating a lot of bandwith and his bandwith was limited.
     
    oseymour, Jan 20, 2007 IP