Car Insurance - RFC search - Myspace Layouts - Mortgage Calculator - Mortgage Articles

PDA

View Full Version : robots/htaccess


Solicitors Mortgages
Apr 21st 2005, 7:06 pm
i have some settings on my .htaccess to ban certain areas,
if i then have a blank"" robots.txt file are those areas still blocked? :confused:
or are these totally different things ??


"" = fully spiderable...like this below
V
V
V
V

User-agent: *
Disallow:

ServerUnion
Apr 21st 2005, 7:36 pm
If you have secured directories, the SE's will not be abe to enter,

sarahk
Apr 21st 2005, 9:21 pm
search engines "read and respect" robots.txt (or not)
htaccess is the law

there's a big difference

gatordun
Jul 29th 2005, 9:45 pm
Use both robots.txt and htaccess!