View Full Version : Can you apply robots.txt to the server
NewComputer
May 31st 2004, 6:29 pm
or do they have to be applied within each website? I don't want certain spiders eating any of my bandwidth. Please advise.
digitalpoint
May 31st 2004, 6:35 pm
It's per site (not per server), no way to get around that.
vagrant
May 31st 2004, 7:14 pm
I don't want certain spiders eating any of my bandwidth. Please advise.
if it is a case of not wanting them on any part of any of the sites at all, then you could possibly ban the spiders, via their ip, from the server
misohoni
May 31st 2004, 7:40 pm
There's an amazing thread on robots and how to use them somewhere on this site.
vBulletin® v3.6.8, Copyright ©2000-2008, Jelsoft Enterprises Ltd.