Hi.... This is what i have in my robots.txt file.....it was created with a plugin i am using....is this sufficient?...do i now need to add anything to this file?...confused....lost...etc... # BEGIN XML-SITEMAP-PLUGIN Sitemap: http://www.greekhomerecipes.com/sitemap.xml.gz # END XML-SITEMAP-PLUGIN thanks tony
Add the following to your robots.txt. It will prevent spiders from spidering the WP code folders, and it will keep your feeds from being indexed, which helps with avoiding duplicate content. User-agent: * Disallow: /wp- User-agent: * Disallow: /search User-agent: * Disallow: /feed User-agent: * Disallow: /comments/feed User-agent: * Disallow: /feed/$ User-agent: * Disallow: /*/feed/$ User-agent: * Disallow: /*/feed/rss/$ User-agent: * Disallow: /*/trackback/$ User-agent: * Disallow: /*/*/feed/$ User-agent: * Disallow: /*/*/feed/rss/$ User-agent: * Disallow: /*/*/trackback/$ User-agent: * Disallow: /*/*/*/feed/$ User-agent: * Disallow: /*/*/*/feed/rss/$ User-agent: * Disallow: /*/*/*/trackback/$ Code (markup): Best regards, Dewald
ok...thanks dewald..... this shows in my header currently.....<meta name="robots" content="follow, all" />.....do i need to delete this if i am creating a robots.txt file? alos...where do i put the above in this file....# BEGIN XML-SITEMAP-PLUGIN Sitemap: http://www.greekhomerecipes.com/sitemap.xml.gz # END XML-SITEMAP-PLUGIN in between the # Begin and # end?....or after all of this?