Hi to all, Is there any drawback if we are not using robots.txt in site. Please don't suggest about work of robots.txt
Hi Amit basically we use the "robot.txt" for suggest to the google crower "do follow & no follow" may be its does effect our site.
I guess there won't be much of a drawback but if you have files/folders on your website you don't want them to access because they are personal, they are gonna be available for the whole world to access. Once they are indexed it might take a while for them to be removed from search engines aswell. Either way, it takes a few minutes to make a robots.txt using a generator, what's the big deal?
Mainly robot.txt is used to used to control how search engines and crawlers do on your site. If you specify restrictions on robot.txt file as which pages to crawl and which are not to crawl, it will be good for search engine. Search engines is not going to visit the specified pages or folders once you included those on noindex or nofollow tag. So that your remaining pages will be crawled soon by search engines and you will get better results for your site.
It depends on what kind of site you run. Public/Private. It also helps keep spam bots away, saving you precious bandwidth.
equal to allow everything one drawback is cannot tell robots about your sitemap, you have to tell each search engine one by one about your xml sitemap