I just added a robots.txt to the root of my domain, but my logs report a few 406: Not Acceptable (encoding) on the robots.txt file everyday. I have validated the file with a couple of robots.txt vaildators and everything seems fine with it. Why do I get these errors in my logs? It might be of interest that I used to get about 50 or so 404 errors/day before I added it, now I get about 5 or so 406 errors/day instead... regards Magnus
Make sure that your robots.txt file is created using a plain text editor like Notepad and that it is saved in ASCII/ANSI format, not unicode or anything else.