Hi, Does anyone know if search robots see robots.txt first or the redirects? We have a robots text that disallows a folder. But there are few files in that folder that people have linked to. So they got indexed anyway. If I do a redirect of those files into a non-forbidden folder, will the crawlers scan them? Thank you.
No. robots.txt first. Example: robots.txt disallow: /redirect/ url: http://www.sample.com/redirect/sample2.html (a 301 to) http://www.sample.com/sample2.html then, no decent robots can see sample2.html
Since the pages have been linked with some other pages as well then the probability is that the crawlers might find them from there and hence can crawl the pages despite the fact the the pages have been blocked from robots.txt. So in case you redirect these pages to some other pages then the probability would be 50-50. If the crawlers are coming through the linked pages then crawlers will scan them other wise it will skip.