Can robotx.txt block specific IPs/domain names?

We are seeing this hit our site in awstats: *removed hostname* (edit>> Thank you for the help, elicitservers)

We aren't entirely sure what it is but we think it is a bot and it is hitting us a lot. We want to stop it. Can this be done via robots.txt? OR do we need to block this at our server?

Any help would be appreciated. Thanks.

 

Blocking it at the server/firewall level would be best, I don't think you can block it in robots.txt. You can however likely block it in an .htaccess file, but as usual at the firewall would be best.

 

I prefer .htaccess file. Its very easy to block several ips with it.

 

robots.txt is just a site preferences file to tell the crawlers which pages you would like them not to visit. Its up to the crawler to honor it or not. In your case, you need to enforce the things not to tell your preference. The only solution to restrict them in .htaccess or firewall. The easies way to do is .htaccess.

 

In CPanel, you can block the IPs using CSF (Config Security Firewall). Firewall would be efficient.

If you do not have access to CSF, then go via .htaccess.
Either you can block by IP or UserAgent ( which is less reliable, since it can be spoofed).

 

I was able to block it with .htaccess by using the following code:

order allow,deny
deny from 123-456-123-456-comcast.com
allow from all

Code (markup):

Of course, the above address is just an example... Thanks everyone.

 

Robots.txt file can't block perticular IPs

 

Yeah... we know. We already figured that out. But thanks for posting your worthless signature links about loan modifications.