1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to Noindex WordPress Categories, Author, Pages and Tags from being searched online

Discussion in 'WordPress' started by Vel, Jul 25, 2013.

  1. #1
    I only want my quality pages such as articles to be indexed by google, and keep others like the Author pages, categories, page number, tags, and archive pages as noindex but dofollow so they do not show up in search or get picked up in search for any of my content.
    SEMrush
    Does anyone know how I can do this? The only thing recommended to me was SEO by Yoast but I installed it to my wordpress just now and the tags and categories are still being picked up in search when I do a snippet search of some of my articles, maybe it takes a while to remove those queries or what?
     
    Vel, Jul 25, 2013 IP
    SEMrush
  2. themes4all

    themes4all Well-Known Member

    Messages:
    662
    Likes Received:
    47
    Best Answers:
    6
    Trophy Points:
    100
    #2
    Sure that will take a While.. you will not the Results immediately... the time that the Bots learn about the new Parameters and understand that they have not to index This pages... so you have to wait till the Search engine bots Re-Crawl your Website.

    Goodluck
     
    themes4all, Jul 26, 2013 IP
  3. winappleworld

    winappleworld Active Member

    Messages:
    139
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    55
    #3
    Go to your control panel of website

    In the Home of your database, you will find a robots.txt file. If not, make one.

    Add this to that file and save -

    --------------------------
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /cgi-bin/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /category/*/*
    Disallow: /trackback/
    Disallow: /comments/
    Disallow: /Tags/


    ------------------------------
    This will help the Google bot from indexing it.
     
    winappleworld, Jul 26, 2013 IP
  4. Arick unirow

    Arick unirow Acclaimed Member

    Messages:
    725
    Likes Received:
    298
    Best Answers:
    30
    Trophy Points:
    500
    #4
    Good decision. By doing this you will gain better rank. SE love useful articles. Low content such as tag/category and the rest will reduce rank. Basically, any low content url will cost rank. With less but useful url available in the index, there is no doubt your site will get better rank.
    I guess you already know about it. You have already answered your own question. Look at your sentences above. You don't need to do anything again. Anytime you deal with Search Engine especially Google, you have to remember "The effect is not immediate". It need times.
    Just set the pages as 'noindex' and you are good to go.
    Snap 2013-07-26 at 19.31.02.jpg
    Here are what you should understand:
    1. Don't block any access to those pages.
    2. Don't use .htacess to block access to the pages.
    3. Don't use ROBOTS.TXT to block that pages.
      SE didn't like any sites which block access to specific pages. Categories/Tags/Authors/Archives are considered as IMPORTANT pages by most SE. Those pages help SE indentify proper structures of the site. It is best to NOINDEX them but ALLOW SE to crawl it. If you didn't want SE to crawl it, you should remove Tags/Categories/Archives/Authors from the wordpress core. :p
      Snap 2013-07-26 at 19.52.31.jpg
      Above is bad example of using robots.txt to block the pages. Only block pages which may not useful for both SE and visitors.
      Don't block access, it is better to NoIndex them.
    The reason is simple, you didn't allow Search Engine Bots to CRAWL it and it will make you lose your rank by doing that. NoIndex is better than blocking access. By blocking access you'll make your site lose some link juices. Just remember, low pages such as tag/categories/archives/author are voting for your site/articles. If you block it, SE wouldn't count the votes.
    Allow search engine to CRAWL those pages but don't allow them to index it. How?
    1. Set NoIndex, Follow => in Yoast SEO Title (looks image above).
    Case solved.;)
     
    Last edited: Jul 26, 2013
    Arick unirow, Jul 26, 2013 IP
    Vel likes this.
  5. winappleworld

    winappleworld Active Member

    Messages:
    139
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    55
    #5
    But for any reason if you re-install a new database and when you got yo re-install Yoast plugin, for the time being, Google will crawl those pages and when you re-install and Noindex them later, it will give them 404 error which aint good too.
     
    winappleworld, Jul 26, 2013 IP
  6. Arick unirow

    Arick unirow Acclaimed Member

    Messages:
    725
    Likes Received:
    298
    Best Answers:
    30
    Trophy Points:
    500
    #6
    OP didn't ask about that.
    By the way, The use of robots.txt has been explained clearly by Google. Use it only if there is no other technique possible.
    To deal with 404, use .htacess and 404.php
    Do you know how to redirect all 404 to specified URL using .htacess, right?
    Or, You can use 404.php to redirect traffic to homepages or specified URL. Doing this will improve rank and help SE Bots.
    Bottom line is, Blocking SE with robots.txt will make their bots unable to crawl the pages and the site will lose link juices. NoIndex them is better because SE will be able to learn the structures of the site and no linkjuices got blocked.
     
    Last edited: Jul 26, 2013
    Arick unirow, Jul 26, 2013 IP
  7. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #7
    Noindexing content does not cause it to return a 404, it is just instructing the search engine bot "you can crawl this content, but don't index it". Nothing else. What you mention is really a non-issue and easily resolved if any content does end up being indexed due to a re-install (use the remove URL option in Google Webmaster Tools).
     
    ryan_uk, Jul 26, 2013 IP
    Arick unirow likes this.
  8. Arick unirow

    Arick unirow Acclaimed Member

    Messages:
    725
    Likes Received:
    298
    Best Answers:
    30
    Trophy Points:
    500
    #8
    A nice technique to block SE from crawling the site. I believe other will find it helpful. However, OP should not block their 'low pages' in order to remove those pages from SE.

    Blocking access is good when we have any pages which have 'password protected pages' or 'sales pages'. It is no good to give access to those pages. Bots didn't buy product, am I right? :D

    I like your post. Other will find it useful too. It just not suitable for OP.
    Thanks for explaining to us. I have hard times writing the sentences to explain it. :)
     
    Arick unirow, Jul 26, 2013 IP
  9. Vel

    Vel Active Member

    Messages:
    316
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    58
    #9

    Wow thanks! Guess what I had blocked category and tags on my robots.txt and this morning I got an error on my webmaster tools saying the sitemap I was using for wordpress the BWP had a problem with the /taxonomy_category.xml. So that means it couldn't index my news articles correctly. So I removed them from Robotstxt now.

    Also is Yoast legit or will it ruin my search? I have site in Google news but my traffic dropped big time and I lost revenue in adsense for past months some-days I don't even earn money.

    So how long does it take for those older articles to disappear in google search the ones with /category/tags instead of the real article.
     
    Vel, Jul 26, 2013 IP
  10. Vel

    Vel Active Member

    Messages:
    316
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    58
    #10

    I am lost here, I have a lot of 404 pages, but I thought those will not bring down my site? Do I hare to redirect every single one of those page? I don't want to use to many plug in because sometimes they fail if they are not updated and they can ruin my site is this true?

    I removed a lot of bad articles and tags and they are showing up 404, how do I remove those in the .htaccess?
     
    Vel, Jul 26, 2013 IP
  11. Arick unirow

    Arick unirow Acclaimed Member

    Messages:
    725
    Likes Received:
    298
    Best Answers:
    30
    Trophy Points:
    500
    #11
    Actually, You don't need to remove Tags/Categories. NoIndex them are more than enough. Based on my own experiences, it will takes 3 days if you were lucky and more than six months to take effects in all variants of SE.
    Here are my thoughts:
    1. If I were you, I'll make Category and Tags become useful by adding Category description and few add on paragraphs (custom description works better).
    2. If you already delete them, use .htaccess to redirect those links. Webmaster tools will tell you what links cause 404. Write down the link and create formula to redirect the links. If it is not possible to use the formula, write down each links and redirect it to new url.
    3. While .htaccess can be used to handle errors nicely and fast, 404.php is another nice addition to overcome the problem. Just redirect all the errors to specified URL or to homepage. It is better to use 301 to homepage.
    All the step above didn't require any plugins. I always use less plugins in my WP setup. Using plugin isn't bad but I prefer to make the code and implement it in the core or theme without the need of plugins. Plugins are recommended when there is special case. However, I recommend everyone to use plugin because it is easier and flexible.
    About Yoast, yes, the plugin is safe and good enough. I think you may need to use Yoast News Sitemap too. It is recommended for any news sites. If you had low traffic, make sure to update the content regularly. I prefer at least two contents each five hours for news site. Any errors in links will cause Bots unable to fetch content successfully and thus cause few drop in traffic. I think many news site got the traffic dropped this month, especially if they didn't update the content each twenty hours.
    Adding Videos and longer but straight content will make bots crawl the sites pretty often. If you still getting problem in your news site, it is better to create another thread about it and let other professional members help you. I'm not that smart in News site or WordPress. :)
     
    Arick unirow, Jul 26, 2013 IP
  12. tvre1ease

    tvre1ease Greenhorn

    Messages:
    70
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    10
    #12
    All in one seo pack lets you do that for each post or page
     
    tvre1ease, Jul 28, 2013 IP
  13. Vel

    Vel Active Member

    Messages:
    316
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    58
    #13
    Okay I am pretty much screwed.

    I used noindexed, follow option on Yoast, now none of my articles show up when I do title search, or content search, it only shows my homepage in search. I just wanted to noindex author, tags, categories, archives, so that my articles will be the only thing to show up, now none of the articles are showing up.

    what happened?
     
    Vel, Jul 28, 2013 IP
  14. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #14
    What's your site's URL so I can fully see what you've done (there's a mixture of good and bad advice in this thread). Meanwhile, check that you only selected noindex for the author, tags, categories and archives only.
     
    ryan_uk, Jul 28, 2013 IP
  15. Enursa

    Enursa Well-Known Member

    Messages:
    263
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    100
    Digital Goods:
    1
    #15
    @Vel... Try to resubmit sitemap on Google Webmasters, and wait for a few days.
     
    Enursa, Aug 2, 2013 IP
  16. Mithchen

    Mithchen Member

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #16
    In my wordpress website, I have dug deep within Yoas plugin to get things done. So it seems easy to me by controlling things overe there.
     
    Mithchen, Mar 14, 2017 IP