I see a lot of these in my access log and was wondering what's causing it. All these different ips are just seeking my robots.txt and nothing else. None of my content or anything. Anyone care to shed a light? Thanks
All search engines check for robots.txt to see if they're allowed to index your site (or which parts they're allowed to, etc). That it identifies as Mozilla/Linux is a little odd, but some search engines identify theirselves with browser user agents to try to avoid being locked out. Though to have no indication that it's a spider would normally only be little script and the like, major search engines would have a means of identification. Perhaps someone has a little search engine script that identifies as mozilla/linux and is rather inefficiently regrabbing robots.txt from a bunch of IPs.