Just read this on boingboing I thought people in this section would be interested. Whitehouse.gov got a new robots.txt, or rather a slim one. 2400 lines in robots.txt, that must have been biggest robots.txt Source: http://www.boingboing.net/2009/01/20/obamas-whitehousegov.html
In white House, earlier there used to be a large Sofa in President office, however, that has been replaced with some kind of tables. When White House is changing everything to welcome its new tenant, Obama, how can website get intact.
Why not? What stops a newspaper from printing the following text? "Many bloggers are reporting that America's official website whitehouse.gov has made some significant changes and more information will be available via search engines; it may look a negligible change but these subtle changes point to many new and big changes that await for United State in future."... robots.txt can also get a small mention. Readers need not be tech savvy to understand the gist
when Indian newspapers could write down about the White House toilet, then why not about robots.txt. After all, we Indians are well aware about these terminology..
you know about robots file because you are in SEO field. ask your friends who are doctors, engineers,lawyers or in other professions...it will be a completely new thing for them. as far as toilet is concerned....everyone knows about it and will read also...this is known as journalism
Good point frih .. I was wondering if there is a site that shows 'historical' changes to various robots.txt files.