If a site had the following robots.txt file what would this mean exactly i have been told this site is disallowing links is this true? User-agent: ia_archiver Disallow: / User-agent: Googlebot Disallow: /frli.htm Disallow: /frre.htm Disallow: /passw Disallow: /ref Disallow: /lin Disallow: /free Disallow: /cgi Disallow: /impressum.htm Disallow: /topframebwinticker.htm Disallow: /send Disallow: /mail Disallow: /ext Disallow: /lin Disallow: /lm Disallow: /sgi Disallow: /bg Code (markup): Any help appreciated
Looks like they don't like Google They are blocking Google spider from accessing those files & directories listed after "User-agent: Googlebot" ia_archiver is alexa's bot. It's blocked for the whole site.