Hello Everyone I work for The Tax Club and I have a question I beleieve this is the right place for me to ask: Is there a limit to the length of a robots.txt file that Google will accept? We are trying to reconstruct a part of our site... Thank you in advance, Best Regards The Tax Club
It seems that Crawlers can read upto 5000 characters in robots.txt. I found this information on several blogs and forums but not sure if Google ever declared this