Robot file for WP

Discussion in 'HTML & Website Design' started by Alanaj, Dec 30, 2013.

0
  1. #1
    I'm putting a new WP site together and want to make sure I have the right info in the robot.txt file. I've pieced it together from searching google. I don't know what the two lines with ?s are for and if they are both needed. Is everything typed correctly and in the right order? I appreciate your help.

    Sitemap: http://www.mysite.com/sitemap.xml.gz

    # For global/any
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /xmlrpc.php
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /*?
    Disallow: /*?*
    Allow: /wp-content/uploads/

    #Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow:/*

    #Google Adsense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow:/*

    #Google Adsense
    User-agent: Adsbot-Google*
    Disallow:
    Allow:/*

    #digg mirro
    User-agent: duggmirror
    Disallow:/
     
    Alanaj, Dec 30, 2013 IP
  2. Hemant Aggarwal

    Hemant Aggarwal Greenhorn

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #2
    Everything is fine, and ?s is for search pages. You will never want your search pages to be crawled by search engines. So dis-allow them.
     
    Hemant Aggarwal, Jan 13, 2014 IP
  3. Alanaj

    Alanaj Active Member

    Messages:
    108
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    55
    #3
    Thank you so much for replying. I had given up on this thread. Now I can move forward confidently. Happy New!
     
    Alanaj, Jan 13, 2014 IP