http://search.msn.co.in/results.aspx?q=site:www.buyrimonabant.co.uk&geovar=56&FORM=REDIR did u guyz ever seen any SE indexing robots.txt ?? check the above link they indexed my robots.txt page.. so weird of them Msn is soo dumb.. plz advice wat to do now??
msn grabbed link to your robots.txt from this forum forums.digitalpoint.com/showthread.php?t=89897 but this is realy weird that they indexed it as usual file.
hey ur right this is strange.. n i want my other pages to b indexed n want this robots.txt to b out of their index.. m not sure why did they did this ??? anyways thnx for ya support
r u sure? i hva to delete the robots.txt . .i thought we have to upload robots.txt and add robots tags in web pages for crawlers to tell to crawl our pages??
By default the search engines index everything. There is no need to specify that. To disallow this or that file or directory is the only reason for a robots.txt. Robot tags are redundant to the robot.txt file and therefore serve no purpose. Here is a typical robots.txt file: User-Agent: * Allow: / Disallow: /cgi-bin Disallow: /db Disallow: /downloads Disallow: /images Disallow: /logs Disallow: /stats
lol, I have never seen anything like that. BTW, you really don't need your robots.txt, if it does no function, just delete it
Yeah, they'll index anything with a link to it. It is really strange though that they would'nt filter robot.txt files out of their search results.