1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to surf as the googlebot

Discussion in 'General Chat' started by tphyahoo, Aug 26, 2004.

  1. #1
    tphyahoo, Aug 26, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Of course that's assuming people are cloaking based on user agent, rather than IP block. :)

    Another problem you will have is sites that look at the user agent to determine if JavaScript will work with their site. For example, you will loose the DHTML functionality at this forum if you are masquerading as Googlebot.
     
    digitalpoint, Aug 26, 2004 IP
  3. tphyahoo

    tphyahoo Peon

    Messages:
    140
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    But my impression is that user agent cloaking is much more widespread than ip block even among the big fish that should theoretically know better.

    But as long as we are talking about this, can anybody vouch for a good ip block service, in case any of us wants to do this? I of course would never stoop so low... :D

    t.
     
    tphyahoo, Aug 26, 2004 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    What do you mean an IP block service? You would need to be physically on a computer that Google uses for spidering to be seen as one of their IPs.

    Now that I think about it, you are going to have even more problems masquerading as Googlebot even just here. Not only will you loose all DHTML menu functionality, but sessions will not work properly. Truthfully, I think you are going to make more problems for yourself than you solve. :)
     
    digitalpoint, Aug 26, 2004 IP
  5. Will.Spencer

    Will.Spencer NetBuilder

    Messages:
    14,789
    Likes Received:
    1,040
    Best Answers:
    0
    Trophy Points:
    375
    #5
    I've been seeing visits to my sites from "Googlebot", but from DSL and cablemodem addresses.

    And, I am not cloaking at all!
     
    Will.Spencer, Aug 26, 2004 IP
  6. tphyahoo

    tphyahoo Peon

    Messages:
    140
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Is a team of people, perhaps shady people operating out of a basement somewhere, that make it their business to compile a reliable list of IPs that google spiders from, and provide this list, for a fee, to webmaster that wish to do ip cloaking.

    Will, that is weird. It could be SEO types that configured themselves to be googlebot and then forgot to turn this off... or it could be some other explanation that I am not creative enough to think of...

    t.
     
    tphyahoo, Aug 26, 2004 IP
  7. rickbender1940

    rickbender1940 Guest

    Messages:
    259
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Maybe competitors seeing if you're up to any funny business? I know I surf some sites in Opera, hit Ctrl-G to wipe out custom formatting and often see attempts to "hide" links with creative colors/placement.

    Makes the H1 redefined to 8pt immediately obvious!
     
    rickbender1940, Aug 26, 2004 IP
  8. Old Welsh Guy

    Old Welsh Guy Notable Member

    Messages:
    2,699
    Likes Received:
    291
    Best Answers:
    0
    Trophy Points:
    205
    #8
    It is fairly common now for people to check out another site by googlebotting. As Shawn said though, the smart money is IP'ing. and unless you are a serious techie, you are not going to set up an IP spoof that will get past them.

    This is why many have siad that high quality cloaking is almost impossible to fathom.
     
    Old Welsh Guy, Aug 26, 2004 IP
  9. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #9
    If you are attempting to see what a page looks like "according to Google", I think the best way is clicking on their cached copy (assuming available).

    It is somewhat humourous looking at all of the User Agents that show up in my web server logs - stuff like "None of your Business" ;-)
     
    hulkster, Aug 26, 2004 IP
  10. tphyahoo

    tphyahoo Peon

    Messages:
    140
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #10
    As I stated (in another thread), I got into this user agent spoofing stuff because I had a competitor who was doing something shady (but interesting!) in a nocache situation.

    I also complained (in yet another thread) that google shouldn't index pages that want nocache, as a measure against SEO abuse. But, google hasn't taken my advice yet, so oh well.

    There seem to be a lot of SEO operations that are savvy enough to
    a) nocache
    b) user agent cloak

    but not
    c) ip cloak

    and in these situations, my little tip on firefox is really dynamite IMHO.

    :cool:

    thomas.
     
    tphyahoo, Aug 27, 2004 IP
  11. tphyahoo

    tphyahoo Peon

    Messages:
    140
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #11
    It would be nice to be able to do this with IE6, and I got the impression from other forums that this was doable with proxomitron. But I am a Dummy, and lazy, and have never used proxomitron, and would like step by step instructions for how to accomplish this, if anybody can post such a thing, or link to where the for dummies instructions already exist...

    t.

    PS: Though actually, I am growing rather fond of Firefox...
     
    tphyahoo, Aug 27, 2004 IP
  12. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #12
    There are reasons you would want to nocache something. For example if you deliver content that is country specific. You wouldn't want users from China viewing cached content that is for US users as an example.
     
    digitalpoint, Aug 27, 2004 IP
  13. tphyahoo

    tphyahoo Peon

    Messages:
    140
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #13
    My proposed Nocache Noindex rule would be a measure against SEO, which hurts google and hurts searchers as well.

    So what if the cache shows chinese for viewers from the US? They could just go to the real web page.

    But google isn't going to do it, mostly because
    a) it could bring them more negative PR from mom and pop websites that grumble, who the hell is google to boss us around like that? elitist bastards

    but also, I believe, because

    b) it would anger some big fish adwords customers that also do agress SEO with IP cloaking and want to keep their seo techniques from being reverse engineered.

    thomas.
     
    tphyahoo, Aug 27, 2004 IP
  14. tphyahoo

    tphyahoo Peon

    Messages:
    140
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #14
    tphyahoo, Aug 27, 2004 IP
  15. Sorvoja

    Sorvoja Don't hide my ads!

    Messages:
    199
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Hi,

    It is not nice pretending to be a search engine spider. I am not even sure if it is legal, if you pretend to be Googlebot then you could hurt Googles goodwill and reputation.

    If I see any fake "googlebots" on web sites I webmaster then it is very possible I ban the ip-addresses used.'

    And yes, I do use the:
    <meta name="robots" content="noarchive" />
    In my experience, very few real users ever look at the Google buffer. Mainly people that are up to no good are looking at the buffer. That is unless I am serving a pdf, doc, or ps file :)
     
    Sorvoja, Aug 27, 2004 IP
  16. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #16
    Would be interesting tphyahoo to see their URL if you can share it - sounds like they are going to a lotta work!

    IP cloaking with Googlebot strikes me as a bit of work, since IP addresses change, and you'd have to stay up with all the ones that are legit from Google ... but doable. Or optionally do a domain name lookup, but that's slows down your web server a bit.
     
    hulkster, Aug 27, 2004 IP
  17. SEbasic

    SEbasic Peon

    Messages:
    6,317
    Likes Received:
    318
    Best Answers:
    0
    Trophy Points:
    0
    #17
    I use the google useragent If am I trying to check out what a cloaked site is using for their software.

    I haven't had any problems so far.

    I wonder how many people actually use the IP based cloaking?
     
    SEbasic, Aug 27, 2004 IP
  18. Sorvoja

    Sorvoja Don't hide my ads!

    Messages:
    199
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Hi alek,

    There is no need to do a domain lookup, you just need to know what ip-blocks are owned by Google, but then you would have to do it for all other engines too.

    Personally I like to use Gigablast to detect cloacking, since very few are cloaking for their spiders. www.gigablast.com
     
    Sorvoja, Aug 27, 2004 IP
  19. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #19
    Yea, but how many IP-Blocks does Google/Googlebot originate from ... and are you willing to stay on top of it? I don't disagree it's doable ... and you could grep out any IP's with Googlebot User Agents that don't match your list, and add those that are new/changed (and block those that are clearly spoofed), but just seems like a bit of work to be nefarious ... but then again, I should not underestimate these guys! ;-)
     
    hulkster, Aug 27, 2004 IP
  20. SEbasic

    SEbasic Peon

    Messages:
    6,317
    Likes Received:
    318
    Best Answers:
    0
    Trophy Points:
    0
    #20
    I wouldn't underestimate them if I were you, just like alek says.

    BTW, maybe this will help

    I tried to put it in the thread but it was too big.

    Click on this link to view a list of the IP addresses I have got.
     
    SEbasic, Aug 27, 2004 IP