quick question about robots.txt

hello!

I plan on using a robots.txt file to block spiders from crawling certain pages. just a quick question since I'm new to this:

User-agent: *
Disallow: /register
Disallow: /login

Code (markup):

from that code above, I just wanted clarification whether it will block those 2 URL's from being crawled or does the code above mean that it would that block spiders from crawling the register and login directories of my server?

thanks.

 

yes, it is fine SE will not crawle those two directories or subpages

 

Well, it should stop bots from crawling bare in mind some bad bots do not respect robot.txt

 

Then just block those bad bots IP address by .htaccess

 

how can I find all good bots for my robots txt. thanks alp

 

I have a list of some spammy bots , you can only block them...rest would be good bots..

 

thanks very much

 

your welcome.....

 

I searhed about spammy bots. people says can't stop spammy bots because they are skipping robots txt

 

yes you are right..some spammy bots don't follow the robots.txt instruction..in this case you can track their IP addresses and block them using htaccess..