Hello DPr's! Hope everyone is well today I was hoping someone could tell me how to write a robots.txt for the following situation. i want to block * bots from certain folders, but I also want to block the WayBackMachine from * So this is my current robots.txt: How can I block the WayBackMachine from the entire site? Thanks go to anyone for a solution. EDIT - What do you think to this?: Wil this block * from the folder mentioned, block WayBackMachine from * and also stop people using WGET and HTTrack to rip my site?
Uh, your post, in it's edited form, is illegible. If you want to block all bots but WayBackMachine from a directory, first disallow the directory from all bots by using the (*) character, and then use this : User-agent: ia_archiver Disallow: Code (markup):
See mine for a good example on how to save some bytes and uphold maintainability. http://www.jabz.info/robots.txt Also, seismic2 missed the slash after the disallow. After editing your robots.txt file, see if it is buggy. I usually use http://www.sxw.org.uk/computing/robots/check.html for validating robots.txt files.