I have been seeing some sites cookies mess up browsers and showing error results. example: A site has a link to B site. (B site is a normal site) But A site has some strange cookies setting. So when click link from A to B will redirect to an error results. So I like to know do cookies also affect bots to crawl these sites? Thanks/ Regards
So when click link from A to B will redirect to an error results. seems like these are link errors - nothing to do with Robots.txt
Hi, I don't think this phenomenon would result in cookie. Site A might use some other techniques like redirection in meta tag. So, those likes would not benefit site B, IMO. Basically, this is not a crawling problem. all the best,