Hi Dp members, Is here any one blogspot expert? I want to know is robot.txt is required for blogspot? Thanks
Robots.txt is required for all websites if you want to restrict some part of your website from crawlers, whether website is hosted over blogspot or wordpress... Generally if your blog is hosted over blogspot, you didn't have access to update your robots.txt, however, blogger automatically upload a default robots file with your blogspot... that looks like: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Sitemap: http://domain.blogspot.com/feeds/posts/default?orderby=updated