Wikipedia founder takes on Google

Thats why google offers support to thoes who need to slow down googlebot for bandwidth/cpu reasons?

Pierce

 

Given wikipedia's traffic levels it is unlikely it would make a significant saveing.

 

I agree there. I got more yahoo bots then the google ones.

 

That is simply not an indication of more crawl traffic from which search engine. It is known google only uses 2/3 ips in total to crawl sites, where as yahoo uses hundreads. As such thats why more bots show up on forums. But watch the google bot, when its busy its never not busy for more than 1 minute.

Pierce

 

It's not a question of load from the search engines themselves. You don't want Google indexing the edit pages because you don't want people ending up there when they're just looking for the article. You also don't want the edit server results competeing with the read only server results.

There's no reason to start anyone off on the bloated MediaWiki software after finding a wiki article on a search engine.

The only time MediaWiki needs (until a better alternative is developed) to be used is when actual article maintainance is being done. When someone hits the edit button.

Otherwise a simple light weight article viewer is more than sufficient and will significantly reduce costs in hardware and bandwidth.

It took me just a few hours to put together Cubia which runs very fast on a slow computer. A dedicated team could easily improve the concept of Cubia to make it better looking and better able to handle the MediaWiki encoding. Google was very slowly going through the MediaWiki mirror of Wikipedia on the same system. Cubia is getting indexed significantly faster. The second release of Cubia is currently in the works. My goal is to fix a number of display and browsing problems.

Since Wikipedia is run by a bunch of non-software people, it's no wonder every problem involves dumping money into resources rather than fixing the source of the problem.

 

Google doesn't index the edit pages. See:

http://en.wikipedia.org/robots.txt

Again see

http://meta.wikimedia.org/wiki/Wikimedia_servers

Most people will never view a page from the database servers.

 

Interesting tid-bit. At the bottom of every request to wikipedia you find

"Served by srv119 in 0.240 secs"

So you can see what server the request came from and how long it took to generate. You have to view source to see it since it's an HTML comment.

 

They can depending on how many times they hit your server. Thats why you can set the crawl frequency if you're using Google's webmaster tools

 

google is no doubt constantly using all its available brainpower to make better search results. it could take a big team to do better, and i think wiki guy's idea is making search tech open source, so everyone can contribute to the get the best rankings.

this should be interesting

mk