1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Too many googlebot hits

Discussion in 'robots.txt' started by weathor, Apr 6, 2010.

  1. tattoos

    tattoos Prominent Member

    Messages:
    1,903
    Likes Received:
    150
    Best Answers:
    0
    Trophy Points:
    335
    #21
    That's good advice. Same as the advice about the robots.txt
    Also, This image is 87KB, and this image is 20KB. I'm pretty sure that you should be able to reduce the size of the images quite a bit without sacrificing to much detail. The second image could be reduced a lot because it has no detail, only colors.

    As you haven't even bothered to create a robots.txt file after the advice was given some time ago, I think that maybe you don't really care... anyway, good luck.

    Cheers
    James
     
    tattoos, May 5, 2010 IP
  2. weathor

    weathor Peon

    Messages:
    148
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #22
    Thanks a lot!!!! I realy dont know what to do (to let google do anything it wants or not). For now I have the bandwidth needed so I wont create a robots txt...

    (Pages crawled per day)
    [​IMG]
    I've seen the crawl rate and the average number of pages crawled per day and its ok its from google
     
    weathor, May 7, 2010 IP
  3. tattoos

    tattoos Prominent Member

    Messages:
    1,903
    Likes Received:
    150
    Best Answers:
    0
    Trophy Points:
    335
    #23
    Yeah, I wouldn't worry about it too much. I sort of have a similar problem for the last 5 months or so. I have a search site with about 120 index-able pages on it, the results pages have noindex tags on them but are not restricted by robots.txt

    My average kilobytes per day used to be around 1000. My average pages crawled per day jumped from around 100 to this...
    ----------------------
    [​IMG]


    It hasn't let up yet. I considered blocking the results pages via robots.txt to see if it helped but have decided against it for now. As long as the sites not slowing down because of it, I guess I can live with it.

    Current crawl stats... (keep in mind that this site only has 120 pages of indexed html and a few small images... the entire site, including results pages weighs under 15MB)
    -----------------------
    [​IMG]


    So don't feel so bad ;)

    Cheers
    James
     
    tattoos, May 7, 2010 IP
  4. 50plus

    50plus Guest

    Messages:
    234
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #24
    You might want to check your raw log files instead of relying on the numbers of your stats package.
    Lately there have been a few scrapers doing the rounds identifying themselves as Googlebot.
     
    50plus, May 14, 2010 IP
  5. weathor

    weathor Peon

    Messages:
    148
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #25
    I ve checked it thanks. Everything is ok and googlebot did an other jump to 15.000 pages the day

    [​IMG]
     
    weathor, May 24, 2010 IP
  6. kevoh99

    kevoh99 Peon

    Messages:
    175
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #26
    That's incredibly unfair to you, lots of bandwidth, I wish I could help, but I know some nice people here on this forum will help you.
     
    kevoh99, May 25, 2010 IP