Help with robots.txt

Discussion in 'HTML & Website Design' started by addlinkweb, Mar 30, 2012.

  1. #1
    my website is running on Wordpress platform and i want to block a page and all its child pages from my robots.txt file

    the format of the url is www.domain.com/sdr

    i want to remove the sdr and all its child pages, what is the syntax of the robots.txt file to disallow this page ?
     
    addlinkweb, Mar 30, 2012 IP
  2. seoalex

    seoalex Banned

    Messages:
    451
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    140
    #2
    User-Agent: Googlebot
    Disallow: /sdr/

    Add the above lines to your robots.txt. This will block all the content and page inside the folder sdr.
     
    seoalex, Mar 30, 2012 IP
  3. gtk29

    gtk29 Member

    Messages:
    519
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #3
    robots.txt only disallows a section of your site from getting indexed by search engines. This is respected by search engines bots and they do not index those sections or pages. Usually, you specify bot name and then section. You can use * for all bots, for example :
    user-agent : *
    Disallow : /sdr/

    But this does not block those pages from being viewed. Anybody can type in manually the address and access those pages. To block the pages or section of a site, you can use htaccess method or just block them from webhost interface (like cPanel).
     
    gtk29, Mar 30, 2012 IP
  4. addlinkweb

    addlinkweb Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    but sdr is not a folder, its a page generated by wordpress and it is coming from mysql database
     
    addlinkweb, Mar 30, 2012 IP
  5. daniel27lt

    daniel27lt Active Member

    Messages:
    109
    Likes Received:
    6
    Best Answers:
    1
    Trophy Points:
    98
    Digital Goods:
    4
    #5
    Hi there, I hope I can help. Within your robot.txt form ad

    user-agent : *
    Disallow : /sdr.html (or what every extension you use) etc. This will dis-allow this particular page from being indexed and searched.

    Note: its best to have the files you want to hide from search engines within folders, then all you have to do is use:

    user-agent : *
    Disallow : /sdr/

    And this will block everything within this folder. I hoped this has helped.
     
    Last edited: Apr 1, 2012
    daniel27lt, Apr 1, 2012 IP