a must for your sites. make your you update them each time you add a page. Otherwise indexing issues!
Robot.txt file is used for two reasons : 1. To not allow some search engine spiders to crawl your site or some pages of your site that you don't want to get indexed. 2. To save your bandwidth and privacy. Search engine crawlers,keep crawling your website and hence wastes your bandwidth and indexes not useful content also. A simple robot.txt file can b made just by writing some lines in it. Just put the directory or file names in / to stop crawler to read that file. User-agent: * Disallow: /js/ Disallow: /ext/ Disallow: /admin/ Disallow: /css/ Disallow: /includes/ Disallow: /admin/ Sitemap: http://www.yousite.com/sitemap.xml
ya one more thing,. the file should be name robots.txt. You can check even facebook for the same thing. Robots.txt is available there ,and robot.txt is not there. http://www.facebook.com/robots.txt
Sitemap: your sitemap goes here User-agent: * Disallow: This should be a simple robots.txt file that might solve the search engines problem of not having a robots file. But unless you want to nofollow some directories in your site, robots file is not much of an issue.
Understand the simple concept... You don't want bots to crawl your admin area or area which meant useful to you but not for Google search result like admin pages.. so just to say bots not to crawl and index these pages you use robots.txt In case if you don't use Robots.txt and use sitemap as said by Mr. Kumar ...is acceptable.. Because even a minor error can make your website go down in Search result.
i read somewhere that a good practise for robots files is to simple insert http://www.yoursite/sitemap.xml into your robots.txt file and upload to your server. This is supposed to encourage bots to find your sitemap and index your pages. Can anyone comment on this?
I had two sites running without the robot text files for a long time. I received a lot of traffic, BUT when I decided to use a robot on my site I was a little concerned. I found that by using a robot to set up links automatically created much more traffic to my site and sales went up. I bought Zeus for setting up links to boost exposure for my site and establish a better page ranking for my sites. You can obtain good traffic by using good quality information on your site for the search engines to find and receive a good position on the search engines but it takes time that way. DO NOT give up on what you are doing as it will pay off in the end. Your Meta tag information would be considered one of the most important information points for the search engines. Should these not be well set your site will not obtain a good ranking. Most say DO NOT place your meta information in capital letters, my experience says, BULL as capital letters show the word to be very important to the site. You must also make sure these words are also in capital letters on the pages to show there importance. You should also use good key word phrases in the meta tag like, SPUN GLASS ROSE. Here you can see I placed it all in capital letters to show the phrase was very important to that page and the site. DO NOT give up as anything one goes after takes time to complete. I hope this has been of some help for you.