Yeah but I think Brett has said that the robots.txt is just one aspect. The real way they are stopping the bots is with cookies etc required for everyone going to the forums. I don't that it will hurt WMW that much for a few reasons: The way the site worked before was very newbie unfriendly and made it look like you had to pay them money to get access to the information, even when doing a random search in Google. They have always been a one-way resource. Since they don't allow linking in posts, it stifles discussion a bit but also keeps traffic from out-flowing. All the while other webmaster resources, including DP, don't have any issue getting links. I imagine a search through all the DP posts would turn up more than a handful of links to WMW threads. So they will still end up getting second-hand traffic from other resources. I don't know enough about the things involved to make an educated guess myself. My first impression, though, is that there have to be other ways to potentially do this without getting into outright cloaking of pages. Apparently the reason why they fell out of google so fast is many people use d the URL Removal tool on them once they posted about the robots.txt change. Google seems very efficient at dropping sites.
Well judging from their Alexa ranking it looks like it has hurt their traffic. http://www.alexa.com/data/details/traffic_details?url=webmasterworld.com Page views per user is up (registered users tend to view more pages), but total users is way down... almost in half.
That is pretty bad. I think he may have under-estimated the percentage of casual traffic that passes through his site. I imagine the same is true here. While there is a large number of regular users, I imagine the core of the traffic is from more casual people. Who probably often first come across such resources via search engines.
I'm sure they knew the amount of traffic he gets from search engines before they did it (they aren't stupid). See this post.
Can you commiserate with him though and understand the reasoning? Do you keep track of how much bandwidth spiders eat up (as a percentage)? Have you ever even -come close- to doing something similar with this forum?
He's cutting off his revenue sources. User agent blocking in httpd.conf will block 90+% of bad robots. You don't need much in the way of CPM to cover bandwidth costs.
Unless Google makes a special exception for him, it's a 180 day wait once you use the Google Automatic URL Removal System.
It's not my forum, I'm not going to tell him how to run it. It's not much (21.5% of bandwidth in this forum goes to spiders/bots)... msnbot is the worst bandwidth sucker though. This month (so far) 10.25GB went to msnbot, 4.86GB went to Gigabot, 4.11GB to Yahoo Slurp, 2.26GB to Googlebot (then a bunch of unknown bots that were under 1GB). Google definitely is the most efficient bot (maintaining self-control on how much to suck down as well as using 1.1 with compression to reduce bandwidth by about 85%). And no... never even crossed my mind to block bots.
Wow. It goes to show that even someone who has been around for years and should know the ins and outs of what to do and what not to do can make a huge mistake every once in a while. Maybe it will be good for them by causing a shift in the signal to noise ratio, but any positive affects will be anomalies. Why would you run an experiment like that on one of your important sites? I know that as webmasters we do a lot of testing and a lot of playing around with little things, but you'd have to be on no sleep for 3 days and drunk to get a site like that removed from google on a whim.
I problem with this strategy is that when regular users drift away there will be no new ones to take their place.
Something is missing in all this. WMW has been around so long + BT must have plenty of good advice from talented people. I cant imagine any of this being a mistake - the idea and possible repercussion must have been considered. Risky, but not a mistake, though it may turn out that way. Im not tech minded enough to work the details though + I dont care Ive never been able to deal with the design/ layout anyway.
me neither...I hate how they make you register for most of the threads..and they randomly give you a password so I forgot it and gave up i'm sure he's thought this through though..
Actually, I don't think he has. If you go over there and read the thread, it's clear that he wasn't expecting this. WMW has been a great resource over the years and I'm sure it will be "fine".