I am very new to this word...ROBOTS.txt How this file is helpful to our website as far as the search engines are concern? Is this helpfull to gain good ranking at search engine? If yes; What important changes we must make, so that we have good ranking? Any other suggestions; we must take care with regards to ROBORTS.txt. Thanks
Robots.txt has nothing to do with pagerank. This is a file that helps you to allow/avoid search engines crawl certain directories of files that you can specify. Learn more about robots.txt here:www.robotstxt.org ·
Thanks for the update. What if; i dont want search engines crawl a particular directory..... and a condition i want search engines crawl a particular directory is there any particluar keywords to make the search engines crawls a particular directory?
bots by default are set to crawl everything in your website unless you specified them not to. to block a directory: User-agent: * Disallow: /directoryname/ hope that helps