1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Where's Googlebot gone?

Discussion in 'Google' started by dkalweit, Mar 4, 2005.

  1. #1
    Hello. I've been watching my stats on about 4 seperate sites, and I've seen almost no hits by Googlebot or even MediaPartners googlebot in the past 3-4 days. This is down from dozens of hits every day on each site, at a minimum. My sites are still in the Google index, many in top positions-- so I don't think I've been penalized in any way(I've done nothing to warrant that, anyways). Has anyone else seen googlebot/adsense bot lately? I want them to find a few changes I've made in the past couple days... :) Thanks.


    --
    Derek
     
    dkalweit, Mar 4, 2005 IP
  2. keywordguru

    keywordguru Peon

    Messages:
    206
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Hmm, very intersting.

    I am still seeing regular activity. The results have definitely been dancing across the board though.
     
    keywordguru, Mar 4, 2005 IP
  3. SEbasic

    SEbasic Peon

    Messages:
    6,317
    Likes Received:
    318
    Best Answers:
    0
    Trophy Points:
    0
    #3
    dkalweit, keep a look out for your page counts...

    I'd bet that you'll start to see a reduction on the number of pages indexed (Using the API)..
     
    SEbasic, Mar 4, 2005 IP
  4. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #4
    I've seen page counts jumping around for weeks now. For the past week or so, they're actually more steady than they have been(and at a decent level)...

    Any idea what this is? It sounds like you've seen it before?


    --
    Derek
     
    dkalweit, Mar 4, 2005 IP
  5. SEbasic

    SEbasic Peon

    Messages:
    6,317
    Likes Received:
    318
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I've seen it on a number of sites I run...

    I can't give you a difinitive reason as to why it's happening (Otherwise I'd not have seen it), but if I were you, I'd keep a look on the API page count, and also check your Coop weight...

    I suspect it may fall (If you're running in the network).
     
    SEbasic, Mar 4, 2005 IP
  6. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #6
    What did you do when it happened to your sites? Did it come back? If so, how long did was it gone?

    I watch my weight and site: counts daily, so I do see the fluctuations. Nothing too horrible yet in the past couple days.


    --
    Derek
     
    dkalweit, Mar 4, 2005 IP
  7. SEbasic

    SEbasic Peon

    Messages:
    6,317
    Likes Received:
    318
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I haven't seen them increase to the number I was getting previously...

    They're slowly becoming interested again, but It's been about 3 months now, so I guess we'll see.
     
    SEbasic, Mar 4, 2005 IP
  8. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #8
    Care to hypothesize as to why? Maybe the site wasn't updated often enough for Googlebot's tastes? Most of my sites don't change daily, unless it's a dynamic change based on user's activity/input.

    The thing that really confuses me, however, is that the Google MediaPartners bot has stopped visiting-- the MediaPartners bot is the easiest, it seems, to get to visit your site-- just put the adsense code up, and it crawls within minutes of getting pageviews... I'm getting over 30k pageviews/day on these sites, and nothing... Very weird... I wonder if Google has a large segment of machines down or something-- maybe my geographical area...


    --
    Derek
     
    dkalweit, Mar 4, 2005 IP
  9. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #9
    Googlebot spidering on my main sites as well as the forums decreased substantially after the Christmas update and Santy worm hits -- initially, I was concerned that the modifications to the forum software designed to thwart Santy might ne having a detrimental effect but of course that wouldn't/shouldn't have any effect on the main site. Since then, I am seeing a slow return of Googlebot and new pages being added to the index. I have no idea how to account for the decrease in spidering frequency or what else is changing with Google.

    Interstingly, I did not see a decrease in visits from the MediaPartners-Googlebot during this time -- in fact, the opposite was true: visits from MediaPartners increased, leading me to ask in another thread here if it was possible that the historical separation between the regular Googleb ots and the MediaPartners bot had been discontinued -- i.e., is it possible that the MediaPartners bot is also returning information for the regular Google database? If so, given that the MP bot evaluates page theme and context (in order to select appropriate ads), does this have any implications for future changes in the way Google will categorize and rank sites?

    Who knows?
     
    minstrel, Mar 4, 2005 IP
  10. crazyhorse

    crazyhorse Peon

    Messages:
    1,137
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Googlebot 3373+18 47.16 MB 03 Mrt 2005 - 12:07
    MSNBot 2864+18 83.54 MB 03 Mrt 2005 - 17:59
    Inktomi Slurp 1506+183 12.52 MB 03 Mrt 2005 - 17:57
    Google AdSense 754+2 6.03 MB 03 Mrt 2005 - 15:35

    Never seen google adsense before ;)
     
    crazyhorse, Mar 4, 2005 IP
  11. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #11
    crazyhorse, what stats package is that? and what is the user-agent string that resolves to "AdSense"?
     
    minstrel, Mar 4, 2005 IP
  12. crazyhorse

    crazyhorse Peon

    Messages:
    1,137
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Well im using awstats i ll download the log files from the server first and then let you u know what the user string is. ;)
     
    crazyhorse, Mar 4, 2005 IP
  13. crazyhorse

    crazyhorse Peon

    Messages:
    1,137
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #13
    66.249.65.147 - - [03/Mar/2005:12:43:46 -0600] "GET /robots.txt HTTP/1.1" 200 5700 "-" "Mediapartners-Google/2.1"

    I think thats the one , never seen it named like this before seperately im my awstats.
     
    crazyhorse, Mar 4, 2005 IP
  14. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #14
    Okay. It's not a new bot then -- it's just how AWStats is naming it. Did you upgrade to a new version of AWStats recently? Previously, they didn't separate Googlebot from MediaPrtners/Adsense...
     
    minstrel, Mar 4, 2005 IP
  15. crazyhorse

    crazyhorse Peon

    Messages:
    1,137
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Well i guess the server admin did. Advanced Web Statistics 6.4 (build 1.812), that is the current version on the server. Still it is weird that my robot text editor software noticed me that it is a new spider, nevertheless its still Google.

    To get back to the main topic. Im running a phpbb forum and yes i have noticed a certain decrease in spider visits since christmas although on a average of a month my site gets spidered by google around the 50k of times.
     
    crazyhorse, Mar 4, 2005 IP
  16. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #16
    Has anyone noticed a drop in non-phpbb forums? I'm wondering if Google is doing something about phpbb forums in reaction to Santy somehow?


    --
    Derek
     
    dkalweit, Mar 4, 2005 IP
  17. Starbug

    Starbug Guest

    Messages:
    89
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #17
    I noticed seriously increased googlebot activity around 2 days ago..

    For me, Usually that happens around the time of the BL Update - Which fits as there was one very such BL update a couple of days ago :)
     
    Starbug, Mar 4, 2005 IP
  18. nfzgrld

    nfzgrld Peon

    Messages:
    524
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Gbot hasn't left my site for more than a few hours since before the begining of the month. It's only the 4th and it's already coming up on 3000 hits from gbot. I did notice yesterday that it slowed down though. It was hitting every 60 to 90 seconds before for days on end, then yesterday it slowed way down. Now it's hitting about every half hour or so.
     
    nfzgrld, Mar 4, 2005 IP
  19. Briank

    Briank Peon

    Messages:
    109
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #19
    googlebot comes to my forum board on a regular basis. On my friends forum board googlebot is almost always there 24/7. Its very cool :)
     
    Briank, Mar 7, 2005 IP
  20. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #20
    This has been SOLVED! My [former] ISP, ValueWeb(http://www.valueweb.com/) finally admits to the problem. They had a router blocking Googlebot, as I suspected and tried to get them to investigate. I am in the process of switching my dedicated servers to a new ISP(Server Matrix - http://www.servermatrix.com/), and I still have a few sites on ValueWeb's network, where I can see Googlebot coming through now. Sadly, my sites have lost considerable ranking, and were in the process of de-listing entirely. ValueWeb has lost a customer, and so far I'm much more happy with Server Matrix overall-- their customer service(both human and automated) is far superior, and their network seems to be far faster and more robust, etc. All for less money per month, at that! I won't miss ValueWeb.

    To all those that find this in the archives or whatnot, pressure your ISP-- there's a good chance it's them if Googlebot and MediaPartners bots suddenly STOP completely...


    --
    Derek
     
    dkalweit, Mar 10, 2005 IP
    Bernard likes this.