http://www.webmasterworld.com/robots.txt check this link out guys, what do you think? Do you think Brett wants his links and the text spidered more often by SE crawlers? Because robots.txt tends to be the most visited file? I THINK IT IS ANOTHER TYPE OF LINK BAITING STRATEGY Any thoughts?
that is strange. Maybe they're trying to see which robots will index the text in the robots file alone.
It's dynamically generated - makes you wonder if he's actually serving the same thing to googlebot et all. It wouldn't surprise me, as he's disallowed all spiders there before - but I've seen plenty of talk about serving up User Agent: * Disallow: / to individual spiders. It does have a vary/user agent line in the response headers, as well as a no-cache pragma.
The ownered banned the search engines to stop them from sucking bandwidth apparently, old news: searchenginejournal.com/?p=2546
why that much writing in robots.txt. Mostly SEO Suggest that don't add any extra and this site...????
Who cares what's in there? He does not want any search engine robots visiting the site anyways. And read the end of the file. IT"S A BLOG.
that is really great find, i will do some research in this, i think he tryed some thing else and showing diff, we all know they are good at seo, so we need to test once.
# As I was messing with the robots.txt for the 1000th time over the last week, # I had a conversation with myself, that went something like this: # # me: Self, can you publish a blog in a robots.txt file? # self: Sure you can. Robots would just ignore the commented lines. # # me: What would be the point? # self: Fun and merriment. Who knows, you could find world peace in here. # # me: Would people still read it? # self: Probably. If they will read blogger - they'll read anything ;-) # # me: Any downside? Any problems with a regularly updated robots.txt? # self: This is another fine SEO test you've gotten us into. # # me: But don't you need to enable comments? # self: Posting an article and then taking comments? Hey Dufus, it is called # a FORUM - and everyone knows we already have one of those! # # me: I love the idea. It has a cool retro BBS feel to it. I almost want to # break out my trusty old 64, fire up a copy of Cnet BBS, and start posting # some zeroday warez dude. lol! # self: I'm going to try it for awhile. Seems like fun for a change. # # The rest is left as an excersize to the reader. # # - bt Interesting
Exactly and its one of the reasons why he is getting good page rank for the file, everyone started linking back to the file when he banned all the search engines.
yo so u feel this is this reverse trick to get backlinks and gain pr.we say say this as negetative publicity.
Nope as m0rtal said he banned the search engines as they were sucking up too much bandwidth and everyone was surprised at his decision. His post on the decision is here and Danny Sullivan described it as a nuclear bomb. Webmasterworld doesnt need to pull a trick to gain anymore exposure just like Digital point doesnt need too and to be honest it would be a very stupid trick to pull just to gain backlinks - PR means nothing in this case.