Mortgage - Loans - 0% credit cards - Free Ringtone - MPAA

PDA

View Full Version : Is it necessary to write a list of my all my pages in a tobots.txt


ecctao
Jan 1st 2008, 4:24 am
I wish the robot can climb all of my web,and there are no secrets in my web.
And wish the SE can site most pages of my web.
Does it will be better for me to list all my pagers in robots.txt?
Or just write "*" or "all"?

r.suhag
Jan 1st 2008, 4:49 am
We write all pages in "robots.txt", but It is not necessary. This file is use, which page of our website we disallow to crawling by Search Engines "robots".

goliathus
Jan 2nd 2008, 8:15 am
robots.txt is mainly if you need disallow access to any crawler. Not allow. If you want to have all sites listed, create sitemap and put a link in robots.txt ...

in this case robots.txt should content only this line:
Sitemap: http://www.example.com/sitemap.xml

How to create sitemap read sitemaps official site: http://www.sitemaps.org

But it's not needed if you have normal html sitemap and/or well linked inner pages (internal links)

ecctao
Jan 4th 2008, 9:28 pm
Thanks for all!

SwapsRulez
Jan 13th 2008, 12:22 am
You can always go for sitemap & write only the below code in /robots.txt

User-agent: *
Disallow:

Cheers mate... Peace!