Hide content and remove from robots.txt????

I'm sure some bot's are looking in my robots and then searching all those folders for email / mail based pages.

If I remove this from the robots.txt how do i then prevent other 'REAL' SE bots from indexing these pages, by simply adding index no follow to the meta tag or other???

Thanks Ian

 

interesting what you are saying
I think <meta name="robots" content="noindex,nofollow"> should work

 

don't put full paths and the robots that are nefarious have no way to find the files you're hiding.

If you have a folder named scripts that you want to hide for example and you have no other folders named scri*, then use the following:

Disallow: /scri

As the robot exclusion does not need a full path, it'll match substrings like this. Although, if you have links to these folders, they may find them on indexable pages anyway, but at least then they have to hit the whole site to do so.

I've seen some people set up robot honeypots at /honeypot and then put the following line in their robots.txt:

Disallow: /honey

Good ones get by, bad robots get trapped beofre they get much further

 

Thanks. I've tried something a little different and it works a treat.

Ian

 

What was that "different thing" you tried? i wish you shared it