This is just to create awareness to people, so that, you can avoid these mistakes. You guys might be knowing already, but, still sharing my mistakes. - I installed wordfence wordpress security plugin, and configured block fake google bot. It almost killed my search engine ranking, it blocked google bot also. After few days, my websites completely disappeared from the result. - When i checked in the webmaster, found full of authorization error. - Completely removed wordfence, and all changes from .htaccess. - Waiting for google bot to crawl again. -Second mistake, permalink finder has default robots.txt 404 option, if it finds 404 for robots.txt, it sends own robots.txt version with disallowing wp-include and wp-content. It added more damage to my sites. Google is not able to render properly my sites, all urls got unreachable error. I hope this may be useful to newbies. Experts, please guide me the steps to make google bot to crawl again?
Yeah that reminds me of something similar I had years ago. Anyhow, have you already verified your website in Google webmaster tools? You can use the 'Fetch as Google' tool and resubmit your site to the index.
Why would anyone wish to block the google bots even if the plugin has the option ? If you want them to not come to your site and dont crawl, then they will not and your site must have been in the unsubscribed mode. Try resubmitting your site in search engine and also add up in the webmaster and get crawled.
Wordfence has a option to block FAKE google bot, That feature looks like blocked the real google bot too. My site has been added with google webmaster, That's how i found the issue.
Ok good, then you can resubmit. I suggest using some htaccess rules to secure your site instead of this plugin. This is an example of what I have been using on apache servers but you can add more specific rules for your site # END WordPress <IfModule mod_rewrite.c> RewriteEngine on RewriteCond %{REQUEST_URI} ^(.*)?wp-login\.php(.*)$ [OR] RewriteCond %{REQUEST_URI} ^(.*)?wp-admin$ [OR] RewriteCond %{REQUEST_URI} ^(.*)?wp-signup\.php(.*)$ RewriteCond %{REMOTE_ADDR} !^xx\.xxx\.xxx\.xxx$ RewriteRule ^(.*)$ - [R=403,L] RewriteRule !^wp-includes/ - [S=3] RewriteRule ^wp-includes/[^/]+\.php$ - [F,L] RewriteRule ^wp-includes/js/tinymce/langs/.+\.php - [F,L] RewriteRule ^wp-includes/theme-compat/ - [F,L] RewriteRule !^wp-content/ - [S=3] RewriteRule ^wp-content/[^/]+\.php$ - [F,L] </IfModule> <Files "\ (wp-config.php|wp-config-sample.php|readme.html|license.txt)"> Order allow,deny Deny from all </Files> <files ~ "^.*\.([Hh][Tt][Aa])"> order allow,deny deny from all satisfy all </files> Code (markup):