Blocking bots

I want to block all spiders and crawlers and programs from accessing a folder on my website using .HTACCESS file

The folder has my product in it.

Users purchase it. Paypal redirects them to the page on my site. This means the people who purchase my product need to be able to view the page when paypal redirects it.

I want to block everything else. All programs and spiders, crawlers, creepys, critters, rollers, back flippers and black magic voodoo trancers.

Who has the code

 

Create a robots.txt file:

http://www.robotstxt.org/robotstxt.html

 

User-agent: *
Disallow: /

Simple as that in a robots.txt file!

 

All they have to do is crawl the site and then read robots.txt.

So I need to use htaccess...right?

Unless there is a way to make robots.txt hidden from spiders 'n crawlers and still function the same.

 

Why would you want to hide robots.txt? The whole point of it is to be read by well behaved robots and control their actions.

 

some people like to keep their private data - well private.

Others do it to help control the flow of say ecomm stuff.

There are many 'valid' reasons one would want a bot NOT to view that page.

 

Because people use bots too. There are thousands of programs.

Teleport Pro, for example.

i don't want someone to be able to run a program and see robots.txt