Hi i want to set Google robots for my new website. Can any one help me how to set Google robots on my website. regards
u first do sum SEO for ur site .. so that they will doo sum keyword analysis, will made sum meta's .. so they these ol r used in robots.txt file
This is right from the Google web masters blog. "You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one)." I wonder how many people have added a robot.txt file and are actually making their seo worse? Blocking the robots from reading all your content? Too funny.
Firstly before to set your robots.txt file you need to know (which I am sure you know) what application or CMS do you use. If you use WrodPress when you can google what dis allowances you need to make for wordpress blog, if you have forum like based on phpbb when you need to google for phpbb forums. The most important thing to remember about robots.txt that the principles of this file is to correct the right paths of your website structure by disabling some access to optimize the crawl operation for search engine.
Before applying anything on your website, it is required to know why it is required for your website. So first learn about robots.txt and its use, then try to add the same in your website