1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

what disallow or allow in a wp blog

Discussion in 'robots.txt' started by frustik, May 17, 2011.

0
  1. #1
    hello,
    i have the blog in second directory of my site:

    i have a blog in a second directory:
    mysite/blog

    i want to be allowed the articles of the blog and of course the first directory "mysite", but I need to disallow some directories into the blog, how to do? i tought:

    user agent: *
    disallow: /blog/wp-admin
    disallow:/blog/search

    is correct or the risk is to disallow also the entire blog??

    thank you
     
    frustik, May 17, 2011 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    the robots.txt code is correct. It would restrict crawlers to crawl admin areas and search results of the blog. however, you would also wish to avoid crawlers to index pages sorted by category, tags, author, date etc as this would lead to duplicate content issue. Now if you restrict them using robots.txt, the crawlers would skip those pages and the inner posts would not be crawled properly, so this doesn't make sense. The alternate way to do this is using Ultimate Noindex Nofollow Tool plugin, this would allow you to place <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW"> on category, tags etc pages so your restricted pages would not be indexed but the inner posts would be crawled regularly. Hope you have answers of your all queries.
     
    manish.chauhan, May 18, 2011 IP