I have 2 wordpress installs on my domain and I have put a robots.txt in the index to stop google indexing certain parts. I got the file of the net and I just want to check it here. My 2 installations of wp are /previews and /sports-betting I had some problems with google indexing the plugins folder before. I am hoping to fix it. This code has a disallow for index.php of wordpress, but I took it out because I thought that google needs to index the index.php file. User-agent: Googlebot Disallow: /previews/*? Disallow: /previews/*.css$ Disallow: /previews/*.inc$ Disallow: /previews/*.js$ Disallow: /previews/category/*/* Disallow: /previews/comment-page/* Disallow: /previews/*/feed/$ Disallow: /previews/*/feed/rss/$ Disallow: /previews/*/trackback/$ Disallow: /previews/wp- User-agent: * Disallow: /previews/archives/ Disallow: /previews/category/ Disallow: /previews/comment-page Disallow: /previews/feed Disallow: /previews/feed/ Disallow: /previews/page/ Disallow: /previews/tag/ Disallow: /previews/trackback/ Disallow: /previews/wp-admin/ Disallow: /previews/wp-content/ Disallow: /previews/wp-includes/ User-agent: Googlebot Disallow: /sports-betting/*? Disallow: /sports-betting/*.css$ Disallow: /sports-betting/*.inc$ Disallow: /sports-betting/*.js$ Disallow: /sports-betting/category/*/* Disallow: /sports-betting/comment-page/* Disallow: /sports-betting/*/feed/$ Disallow: /sports-betting/*/feed/rss/$ Disallow: /sports-betting/*/trackback/$ Disallow: /sports-betting/wp- User-agent: * Disallow: /sports-betting/archives/ Disallow: /sports-betting/category/ Disallow: /sports-betting/comment-page Disallow: /sports-betting/feed Disallow: /sports-betting/feed/ Disallow: /sports-betting/page/ Disallow: /sports-betting/tag/ Disallow: /sports-betting/trackback/ Disallow: /sports-betting/wp-admin/ Disallow: /sports-betting/wp-content/ Disallow: /sports-betting/wp-includes/ Code (markup):
Hi. There's quite a lot of duplication there, and a lot of things are being blocked that I don't think are necessary. Personally I would just go for: User-agent: * Disallow: /previews/wp-admin/ Disallow: /previews/wp-content/ Disallow: /previews/wp-includes/ Disallow: /sports-betting/wp-admin/ Disallow: /sports-betting/wp-content/ Disallow: /sports-betting/wp-includes/ Code (markup):
I used that robots.txt and I tried to remove my plugins directory from google using the webmaster tools "remove url" feature, however it says: "Your request has been denied because the webmaster of the site hasn't applied the appropriate robots.txt file or meta tags to block us from indexing or archiving this page."