I'm wondering if there is a size limit for the robots.txt file. The reason I'm asking is that I have a large number of individual image files & background files in my root directory that would be a real pain to put into one folder because I would have to update all of the links manually. But if I exclude all of them one by one my robots file would be ridiculously large. Is it possible to just exclude all JPG files or do I have to exclude each one individually? Any help you can provide would be great! ]
well brett tabke's blog is currently about 26k, hehe. Most of it is commented anyway. http://www.webmasterworld.com/robots.txt