Help needed - robts.txt and sitemap

visit vision fez's sites they explain it in a good fashion

 

Thanks for this answers i think this will also save your Your Bandwidth.

 

It is important

 

robots.txt
just see and learn the example of others site robots.txt

which part u are not allowed to index in search engine so put in disallow path
that is simply easy

 

I think, if you don't want to control spiders then there is no need to use robots.txt, for sitemaps xml is must and useful.

 

Does disallowing access to certain folders make those folder more susceptible to hack attacks from bad guys.

 

If i want every page of my site to be visited by Google and Yahoo, does it need to have robot.txt?

 

you can find complete references about robot.txt in google

 

Forget robots.txt file. It is nothing important.

Learn more on sitemap specially if you site have thousands of pages. make more than one sitemaps if needed.

I guess I am bit late to reply here.

 

Robots.txt is cool to specify which parts of your site should be indexed and which ones should not be