please help - wordpress robots.txt

Discussion in 'WordPress' started by burumcmtop66, Jul 2, 2011.

0
  1. #1
    okay so i have a home remedies blog, google has index all pages including the admin folder. i wanted to know which folders to include in the robots.txt files to disallow bots to crawl them?i just want google to index the main home page, post pages and other pages.

    hope someone can help me
     
    burumcmtop66, Jul 2, 2011 IP
  2. aap

    aap Well-Known Member

    Messages:
    1,802
    Likes Received:
    39
    Best Answers:
    2
    Trophy Points:
    120
    #2
    Add following lines in robots.txt 

    
User-Agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
    Code (markup):
     
    aap, Jul 2, 2011 IP
  3. tricks2

    tricks2 Active Member

    Messages:
    427
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    78
    #3
    i use the following robots.txt it can be help for you...

     
    tricks2, Jul 2, 2011 IP
  4. oliva

    oliva Member

    Messages:
    79
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    48
    #4
    Thank you, I had the same question. Will it help to get rid of ?replytocom=xxx?
     
    oliva, Jul 4, 2011 IP
  5. тнє Sufi

    тнє Sufi Peon

    Messages:
    391
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Yes it will help.
     
    тнє Sufi, Jul 6, 2011 IP
  6. chromoz

    chromoz Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Nope..!! If you have set canonical URL ...and set replytocom as noindex in URL paramater of GWT..That wil help...
     
    chromoz, Oct 5, 2011 IP