How would a robots.txt look like if all it had to do was block bots from visiting this one page and the threads on that board? I get a lot of spam there and I want to make sure the bots don't crawl it before I have a chance to delete it. The page would be something like http://www.xmysitenamex.com/forum/viewforum.php?f=4 Also, would I put the robots.txt right in my public_html or public_html/forum? Thanks
robots.txt only works on robots that obey it - unlikely to do anything about your problem. public_html - root directory. Best bet - move to something that isn't as easily hit by bots. If you are on IPB move to 2.2. If on phpbb...find something else (FUD is great).
Thanks davert, but I wasn't talking about stopping spam, rather stopping SE bots from seeing those links.