Hello all there using digital point forums here is one query in my mind regarding robots.txt file about my website www.vocso.com. I want to know how it can be beneficial for a web site and it's proper way to use. My robots.txt page is http://www.vocso.com/robots.txt this please suggest it is good or should be added more. Thanks & Regards, Alisa Rozzy
Robots.txt files inform search engine spiders how to interact with indexing your content. * By default search engines are greedy. They want to index as much high quality information as they can, and they will assume that they can crawl everything unless you tell them otherwise. * If you specify data for all bots (*) and data for a specific bot (like GoogleBot) then the specific bot commands will be followed while that engine ignores the global/default bot commands.