Hey guys, I just would like to discuss with what benefits can be gained from robots.txt file when promoting website? Could you please explain how to upload this file to hosting? Is it uploaded to public_html root directory as well?
Yes, upload it to your domain's main directory. Personally, the main reason I use a robots.txt file is because I use WordPress for all of my sites, and there are a lot of extra pages that don't need to be indexed (such as archives, log-in pages, etc).
The main functionality of robot.txt is to block search engine spiders from indexing certain web pages of your website. Some people use in case were they have the same content on various web pages and therefore avoid duplicate content issue. You can either add robot.txt in the meta field of the web page you do not wish them to crawl. It can also be be added in the root directory as a robot.txt file.
upload it in the root directory of your website so that it can be accessible through url www.yoursite.com/robots.txt
u can also put your sitemap location in your robots file to inform search engines of your sitemap http://www.sitemaps.org/protocol.php#submit_robots