Hello, I have mydomain.com/blog and I have wordpress installed in the blog folder. I was wondering if anyone can take a peek at my robots.txt and let me know if this is ok for having my blog in a sub folder? I am not sure if I should "Dissallow" some of the things I am dissallowing. The other issue I am concerned about is I am using a dynamic sitemap for the blog so I have added another line for that in robots.txt. Thanks for any help, User-Agent: * Disallow: /privacy-policy/ Disallow: /terms-of-use/ Disallow: /folder1/ Disallow: /folder2/ Disallow: /missing.html Disallow: /blog/wp-admin Disallow: /blog/wp-includes Disallow: /blog/wp-content/plugins Disallow: /blog/wp-content/cache Disallow: /blog/wp-content/themes Disallow: /blog/trackback Disallow: /blog/comments Disallow: /blog/*.php$ Disallow: /blog/*.js$ Disallow: /blog/*.inc$ Disallow: /blog/*.css$ Disallow: /blog/*.gz$ Disallow: /blog/*.wmv$ Disallow: /blog/*.cgi$ Disallow: /blog/*.xhtml$ Disallow: /blog/*?* Disallow: /blog/*? Allow: /blog/wp-content/uploads Sitemap: http://www.mysite.com.com/sitemap.xml Sitemap: http://www.mysite.com.com/blog/sitemap.xml
I think, your robots file is okay no worry about that, you don't block your content, everything is okay.