Cloak Probing and User-Agent spoofing

[Sholva]

Cloaking is an old trick (occasionally there are legit reasons for it) but obviously it can be used to try and fool search engines.

Do you think Google and other crawlers do sneak testing with a common User-Agent to look for gross differences occasionally? I suppose tricky cloakers could check the ip range but even then they could miss some ranges that are in the hands of Google.

People always are quick to say that google doesn't like cloakers so don't do it... But how would you feel if you found out that a search engine was spidering your site under a MSIE 6.0 user-agent? Would you be bothered or couldn't care less?

Just curious to what everyone has to say on the matter. Thanks

[SEbasic]

If I were to Cloak ;) I would use IP based cloaking so it wouldn't happen (Unless an engineer were to check at home).

[Sholva]

If I were to Cloak ;) I would use IP based cloaking so it wouldn't happen (Unless an engineer were to check at home).

This is true, but they could have ip ranges that you don't know about / and don't resolve with "google" or are owned by a 3rd party so a who-is won't help either.

Also, what if joe-bloggs wrote himself a crawling spider and decided he didn't trust webmasters and made the entire crawl announce itself as MSIE? Would you be peeved if you found out? Personally I don't think I'd care unless it was an aggressive bot, I think others would care though.

[SEbasic]

To be honest, if one site goes down, I personally would not be all that bothered...

Google has publicly stated that they will normally (I understand that this isn't always the case), make algo tweaks to filter out results rather than remove them manually...

A cloaked site is so easy to put up (so I hear), that it only takes 10 minutes to put up a replacement...

If the algorithms change, then so be it - back to the drawring board...

But that's part of the fun anyway :)

Also, what if joe-bloggs wrote himself a crawling spider and decided he didn't trust webmasters and made the entire crawl announce itself as MSIE? Would you be peeved if you found out? Personally I don't think I'd care unless it was an aggressive bot, I think others would care though.
Then really - there's not a lot a webmaster could do... Code hijacking is a pain but it happens and there really isn't a lot you could do...

[vlead]

According to me there is no fool-proof way of cloaking... and there are no fool-proof search engine algos either.... ;)

[PRBot.Com]

Yeah, spiders can also be gived a few 1000 proxies to hide behind :D

[nadlay]

But how would you feel if you found out that a search engine was spidering your site under a MSIE 6.0 user-agent? Would you be bothered or couldn't care less?


If you have nothing to hide, why would you be bothered?

[Sholva]

If you have nothing to hide, why would you be bothered?

I'm inclined to agree, I was considering testing a small scale spider I've been writing and I wanted to know what people would think if a % of the spider activity was hidden behind a common browser user-agent, not all though of course. That was one of the ideas I'd considered was browser specific pages seeing how many domains send back customized browser code. But that wasn't the main objective.

I think as long as a spider doesn't overwelm system resources and hog excessive bandwidth for site owners then a few page hits with page user-agents isn't going to hurt. I doubt it would even effect their stats much either.