View Full Version : Is this a valid robots.txt file?
HomeRun
Mar 17th 2008, 6:16 pm
User-agent: *
Disallow:
Sitemap: http://www.mysite.com/sitemap_index.xml.gz
Will it allow all crawlers to every part of mysite?
hans
Mar 17th 2008, 6:28 pm
here you find all you need about robots.txt
http://www.robotstxt.org/
read carefully !!
your "Disallow:" has no path - hence it would be WRONG code anyway most likely
and if interpreted wrongly by some bots it may do just the opposite of what you want
disallow ALL
HomeRun
Mar 17th 2008, 7:11 pm
So can I remove the disallow line and also the user agent line if I don't want to block any bots for now?
or maybe just the disallow line
Edit:
here they said no path will allow all bots to my site http://www.robotstxt.org/robotstxt.html
To allow all robots complete access
User-agent: *
Disallow:
then
To exclude all robots from the entire server
User-agent: *
Disallow: /
hans
Mar 18th 2008, 1:37 am
yes you may have an empty robots.txt
but keep in mind that there might be folder you never want bots to be in - like entire
/cgi-bin
and
any administrative folders
anilwebmaster
Apr 1st 2008, 4:02 am
Thks for you answers.....
I had question .... whr to insert this file... and whr to show in the source code..?
hans
Apr 1st 2008, 4:50 am
robots.txt
always is in domain root /
there is no need to link to it
search engines always search for robots.txt at
yourdomain.com/robots.txt
by default
manish.chauhan
Apr 4th 2008, 8:41 am
If you want to allow all bots to crawl your all website, then there is no need to add robots.txt file..
vBulletin® v3.6.8, Copyright ©2000-2008, Jelsoft Enterprises Ltd.