1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Why is Google crawling so much?

Discussion in 'Site & Server Administration' started by hmansfield, May 21, 2007.

  1. #1
    Google and yahoo are crawling the crap out of my new blog, and driving up bandwidth.
    Googlebot 21331+98 1.04 GB
    SEMrush
    Is this normal? I have to set aside 2g worth of bandwidth just for google?

    Anybody know if this is normal?
    What are you guys getting crawled at?
     
    hmansfield, May 21, 2007 IP
    SEMrush
  2. nukepuppy

    nukepuppy Peon

    Messages:
    93
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #2
    you postd the same thing about yahoo..

    you should look into sitemaps? they're not crawling to index.. they're crawling to find what to index...

    if you're already using sitemaps just give up on the internet.. and remove disallow robots ;P


    I get crawled pretty much all day by google... doesnt matter though..
     
    nukepuppy, May 21, 2007 IP
  3. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #3
    I thought I had sitemaps activated.
    It's only these 2 ,(Google and Yahoo, especially yahoo) the other bots seem to be normal.
    Can you elaborate a little.
    It shows that Google has over 4,000 pages indexed, what are they still looking for?
     
    hmansfield, May 21, 2007 IP
  4. coolguy27

    coolguy27 Banned

    Messages:
    485
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #4
    It is normal if you always update your contents. Mine also being crawled by Google whenever I updated my blog's new contents...Don't worry about it... Be proud that your site is always visited by Google... :D:D:D
     
    coolguy27, May 21, 2007 IP
    hmansfield likes this.
  5. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #5
    But the bandwidth! Three different urls from yahoo are hitting me for 2g's this month and another 1.5g's from Google. I know that is not normal..is it?
     
    hmansfield, May 22, 2007 IP
  6. PinoyIto

    PinoyIto Notable Member

    Messages:
    5,863
    Likes Received:
    170
    Best Answers:
    0
    Trophy Points:
    260
    #6
    I guess that is normal, specially if you have lot of backlink from reputable sites. If bots have lot of way to go to your site they will visit it frequently. 2GB BW is not that high, I have a entertainment site getting more than 10GB BW from google and I love it because everytime I update my site google index it immediately and sooner or later start getting traffic from that new page..

    Like other says be proud if google bot loves your site....
     
    PinoyIto, May 22, 2007 IP
    hmansfield likes this.
  7. just-4-teens

    just-4-teens Peon

    Messages:
    3,968
    Likes Received:
    168
    Best Answers:
    0
    Trophy Points:
    0
    #7
    2GB seems high.

    mine are - Googlebot 9311+41 35.71 MB

    9311 x 3 = 27933 (that would work out more hits than yours but would only work out to be 100mb bandwidth)

    is google trying to download files from your site (.zips, .mp3s etc?)
     
    just-4-teens, May 22, 2007 IP
  8. Forrest

    Forrest Peon

    Messages:
    501
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    0
    #8
    If you have 4,000 pages in their index, maybe. I wish they crawled my site half as often!
     
    Forrest, May 22, 2007 IP
  9. Kawish

    Kawish Active Member

    Messages:
    446
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #9
    it's normal.
     
    Kawish, May 22, 2007 IP
  10. imujval

    imujval Peon

    Messages:
    38
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #10
    you can set duration for spider throught robot tage. how many day after it will visit ur site or u can manually ping to google and yahoo when u update ur site if u are not updating ur site soo much frequently.
     
    imujval, May 22, 2007 IP
    hmansfield likes this.
  11. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #11
    Thanks guys. I am only going on the word of one of my associates that is telling me that this abnormal activity.
    I though the same way you did, that it is good, especially with the amount of content that I am adding, and having the pages translated with a global translator.

    None of that spider activity counts as stats, ie: page views correct?
    Just hits, right ? Or not?
    It is still very new, and the page views compared to visits, are high.
    This is my first blog of this size, so it all looks strange to me.
    Thanks for the help.
     
    hmansfield, May 22, 2007 IP
  12. MatthewN

    MatthewN Well-Known Member

    Messages:
    859
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    195
    #12
    I wouldnt worry about it. 2Gb isnt a lot of bandwidth and only costs a couple of dollars a month at most.
     
    MatthewN, May 22, 2007 IP
    hmansfield likes this.
  13. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #13
    Yeah, I think I may be getting some less than expert opinions from my associate..
    Thanks
     
    hmansfield, May 22, 2007 IP
  14. nxb22833

    nxb22833 Banned

    Messages:
    50
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #14
    I would not complain if I were you. It is a good sign to come
     
    nxb22833, May 22, 2007 IP
  15. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #15
    I don't have anything to download, most of the links go to an affiliates site for downloads, (Music)
     
    hmansfield, May 22, 2007 IP
  16. just-4-teens

    just-4-teens Peon

    Messages:
    3,968
    Likes Received:
    168
    Best Answers:
    0
    Trophy Points:
    0
    #16
    i wouldnt worry then, probably just down to the amount of pages u have and how big each page is etc.

    how does yahoo/msn etc compare to google? do they use as much (in equivalent)
     
    just-4-teens, May 22, 2007 IP
  17. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #17
    Yahoo is sending bots from 3 different URLs at about 2.5 G's this month.

    216.109.121.44 l 12663 l 12663 l 437.80 MB
    216.109.121.41 l 12652 l 12653 l 443.23 MB
    216.109.121.42 l 12474 l 12474 l 436.16 MB

    MSN, not so much and only 20 pages indexed [545+99]
     
    hmansfield, May 22, 2007 IP
  18. inworx

    inworx Peon

    Messages:
    4,862
    Likes Received:
    201
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Google, I guess has got lots of dirt cheap bandwidth. Google bot are crawling and using more bandwidth then used by other methods through search engines...
     
    inworx, May 23, 2007 IP
  19. hmansfield

    hmansfield Guest

    Messages:
    7,904
    Likes Received:
    298
    Best Answers:
    0
    Trophy Points:
    280
    #19
    Yeah, after I tracked down the Yahoo IPs, now it seems that all is well.
    Those 3 Yahoo addresses that were eating so much bandwidth?
    After further investigation, it seems that they are each 1 individual IP address, but they are the main addresses for Yahoo ISP's all over the world, like SBC.
    So each one is responsible for hundreds of Yahoo DSL IP's.

    I am no longer freaking out, and it seems that I have traffic:)
    I never handle server admin, so this is all new to me.
    I have a list if anyones interested.
    Thanks for all the help guys.
     
    hmansfield, May 23, 2007 IP