How to "block" specific URLs to not get crawled?

freelance deziner Member

Messages:: 270

Likes Received:: 1

Best Answers:: 1

Trophy Points:: 40

#1

I run a wordpress blog and it has a voting plugin installed, the problem is that the voting button using Ajax technology and when Google crawl my site it looks for the "voting" URL too which doesn't exist, therefore Google showing error message in Google Webmaster Tools that the no. of Not followed pages have increased.

The links are like - wp-admin/admin-ajax.php?action=wti_like_post_process_vote&task=like&post_id=6571&nonce=9e0ce65073

I want to ask that is it possible to "block" wp-admin/admin-ajax.php and all links after that so that Google should not go after them?

Or may be some other option you suggest?

freelance deziner, Sep 26, 2013 IP

competent123 Notable Member

Messages:: 1,752

Likes Received:: 71

Best Answers:: 6

Trophy Points:: 255

#2

The noindex meta standard is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.

To prevent all robots from indexing a page on your site, place the following meta tag into the <head> section of your page:

<meta name="robots" content="noindex">

To allow other robots to index the page on your site, preventing only Google's robots from indexing the page:

<meta name="googlebot" content="noindex">
When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it. Other search engines, however, may interpret this directive differently. As a result, a link to the page can still appear in their search results.

Note that because we have to crawl your page in order to see the noindex meta tag, there's a small chance that Googlebot won't see and respect the noindex meta tag. If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. (Also, if you've used your robots.txt file to block this page, we won't be able to see the tag either.)

If the content is currently in our index, we will remove it after the next time we crawl it. To expedite removal, use the URL removal request tool in Google Webmaster Tools.
Click to expand...

https://support.google.com/webmasters/answer/93710?hl=en

competent123, Sep 26, 2013 IP

LinkGenie Active Member

Messages:: 794

Likes Received:: 8

Best Answers:: 0

Trophy Points:: 60

#3

You can use

<meta name="googlebot" content="noindex">
Since you are using Wordpress, create another header php for to include for no-index pages.

LinkGenie, Oct 1, 2013 IP

Log in or Sign up

How to "block" specific URLs to not get crawled?

freelance deziner Member

competent123 Notable Member

LinkGenie Active Member

Useful Searches