Just thought of a quick one.. Trust me when I say this isnt a "My friend has a website and..." story.. Its not related to myself but could be! :0) If you are blocking Google ( and other robots) from scanning certain folders on the pass of your site and say you're using a "Gray" hat tools ... Whats stopping them from snooping into certain folders with certain keywords that they could flag? They could visit the folders manually , or base their blocking algos on folders in with certain "black" words... Anyone any theory on this?
If you have told them not to index a folder THERE'S NO POSSIBILITY OF HAVING BLACK HAT SEO THERE. Since the pages wouldn't be indexed, any SEO will have no benefit.
Not actually worried about them getting to any folder but whats to say they wont use the dissallowed commands in robots.txt to actually have a peak in there..?
Not a damn thing...especially if there is a link anywhere to it... In fact there are quite a few crawlers that will not respect the instructions for robots. You have to remember however most people who try to blackhat are those that cannot obtain results via whitehat grayhat bluehat zydeco hat too many hats....
I believe Baidu is currently ignoring robots.txt. It's a good point; Google could in fact use that file as a guide to all the less-than-reputable things your site might be trying to do.
Yep and hence why i do indeed wonder.. Especially now , they wont let you buy paid links... They're getting bigger and bigger and bigger for them boots.. Where is the romance in it ?
I don't think they have time to mess around with things like that, there is far too many white hate reasons for using robots.txt for them to find a way to penalize you for blocking them on certain pages. Although someone from Google will 'prolly see this thread and work it into the next algo update In short it's a good idea but no, I don't think it could be right.
They won't let you buy paid links? No that is not true....you can buy links for traffic purposes. However if you buy links to effect PageRank or SERP positions then Google will devalue the links. But then the question is can Google determine all of the INTENT in purchasing links and the answer there is; NO Lastly every webmaster should remember that it is more important to build a business online.... than to worry about what the thoughts are of a search engine.
Sorry not saying they "wont" let you buy links.. Just wondering how Google would figure out the Intent :0) Thanks for the feedback!
Knowing how low Google is stepping these days with other business practices, I would not consider this far-fetched at all.
robots.txt is only a gentleman's agreement, nothing stopping a bot from crawling disallowed folders. I still don't get the benefit of that. Do you really believe someone is sitting there taking a look at disallowed folders of millions of URLs?