IA Archiever hits my website several time a day.What is it? If it is something like archiever then why it is archieving my website and who has granted it permission to archieve my website? How i can block it to visit my website? I am looking forward for your answers Thank You.
Why block a robot that can be useful? It archives your website for "history" purposes at www.archive.org and it's used for data to www.alexa.com
How is it useful? People no longer credit Alexa as an ranking authority.And it is archieving website without any permission. Is there anyway to block it?
Google also "archive" your site, and they didn't asked for your permission. If you want to block it, add this to your robots.txt file (or create it if you don't have it yet, put the file in your root of your domain) User-agent: ia_archiver Disallow: / Code (markup):
I am not sure how to react to new situation. Currently my robots.txt is User-agent: * Disallow: What it should be ?
It you want to ban only archiver, robots.txt should be: User-agent: * Disallow: User-agent: ia_archiver Disallow: / Code (markup):
SE don't archive pages forever, like archive.org does, but SE usually allow you to see the cache of the pages of your site that they have crawled for SERPs purposes, for example: http://216.239.59.104/search?q=cach...ey-directory.com+inurl:Bands&hl=en&lr=lang_en
Depending on your site size you might just want to block them by using .htaccess. I have heard about ia_archiever going thru sites before finding your robots.txt file. It is very easy and can be done with just a few lines of code added to your file. You can do this by ip address or user agent. You cou8ld do sonething like this: Have a nice day.
i've got a site online since mai 05 (www.wordconstructor.com) and ia_archiver visits it regularly but i'm still not listet in archive.org. How long does it take to get in there?