WebmasterWorld Out Of Google

I believe the bad bot story is a smoke screen myself. Have you ever known a bad bot to obey a robots.txt file? I haven't. Most don't even check for it. We all deal with them everyday in our own way. I certainly am not going to ban good search engines because of the bad. Why would anyone in their right mind think that a bad bot would obey a complete ban robots txt file when they won't obey the one directed right at them in the first place? This whole story is not to be believed. Brett is NOT stupid and he knows that bad bots don't obey robots.txt and never have. I've been at the game since 1996 and there is more here than meets the eye. If you have bad customers, you don't close your whole business down because of the bad one, you find way to eliminate the bad apples and cause an little damage as possible to your good customers.

Brett did change ISP's recently because of a router crash in Chicago. He may be, for the first time, paying over bandwidth charges which might be catching his attention for the first time. There are honest ways to make money off a good forum that gets this much traffic. However, he might not want to admit that he's scared to death facing these extra bandwidth fees for the first time. Everything we are talking about here is speculation, but what's going on over at WMW isn't speculation, it's real life for folks like me.

 

I still believe that the truth in not the bandwidth question, but something that happened at Pubcon. I heard that Brett personally interview Matt Cutts in Las Vegas during one of their sessions and asked him pointed questions about some of Googles practices including the so called "Sandbox" effect, which Cutts said didn't exist. Most Webmaster do believe that something like a sandbox does exist. I don't think this is the reason, but something happened there that didn't go well for Brett and/or WMW. We may never know, but I have found that I'm spending more time here now in this forum which isn't a bad thing!

 

I guess it's an attempt to lower bandwidth usage. Looks like it's been working!

I don't know- maybe there's more to the story. Brett's been around a long time.

 

Any how, I'm out of this thread. I said what I had to say and now I'm moving on to other more profitable things. Best of luck, Brett!

 

Yea it sucks when SEs eats up bandwidth..

I wish there was a way to limit the number of spiderings in robots.txt
etc.. max 100 per SE

 

Another bad move. That's way overpriced. maybe in the 99 cent bargain bin but no one is going to pay $9.95 for that...

Too long. He's become a tail that thinks he can wag the dog.

 

Seems like good way to get free publicity, if nothing else, to me. Worked like a charm so far.

 

^^ haha yeah probably

 

I'm a little late reading this, but I wanted to also express my shock and dsbelief. Just last month they moved away from Westhost and got a dedicated server. Surely, this new server can handle a few roque bots hitting them, and surely large bandwidth hogs can be properly dealt with. I think somehow Brett has gotten too obsessed in some way. It seems crazy to me.

If he needs a little better server, a better firewall, and 2 or 3 people to help manage traffic, etc., all he needs to do is sell a few ads on his site. It makes no sense. I'm wondering if he somehow is scared of success, and subconsiously does things to prevent success.

 

is it a way of purging googles index?

if he changes the robot text back to what it was surely they would get spidered normally??

and he would go straight back to his normal PR..... or greater (seeing how many more links this has generated.)

or can google (calling his bluff) turn round and say "No, you are out, stay out?"

 

I believe Google makes a domain wait at least 180 days before reindexing after they have been removed. In that time period considering the competition from places like this forum, they might be hurt more than any new linking strategy would help. Time will tell.

 

excuse my ignorance on this issue, but is that official?

 

I really don't think anything we know is official Even the official stuff is up to interpretation lol

 

http://www.google.com/webmasters/remove.html

Apparently, some helpful webmasters submitted the WMW url to googles removal tool after the site owner changed the robots.txt file, so they are out for at least 180 days.

 

well google simply decided webmasterworld is of little value and i tend to agree with google on this

 

er, did you read the page linked to in the first post??!!! They took the site out themselves using robot.txt and the form Google provides.

 

I would imagine he is blocking bots to find the ones that ignore the robot.txt and then ban them through an IP ban which will solve a lot of the bandwidth problems.

Seems like a sensible idea although he is going to lose a lot of traffic doing it.

 

Really? It sounds like a really dumb idea to me. If this is the only way he could think of to figure out who are the "bad bots" so they could be banned, I'd be recommending that he hire someone from this forum to do a bit of consulting.