is am i right with robots.txt file

Discussion in 'robots.txt' started by i_am_dhaval, Jul 11, 2011.

0
  1. #1
    One of my site i am using this
    
User-agent: *
Disallow: /subfolder/
    Code (markup):
    is this mean all pages are allow from the site only pages which is in " subfolder " that is not allow?

    am i right?
     
    i_am_dhaval, Jul 11, 2011 IP
  2. dir18

    dir18 Active Member

    Messages:
    256
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    65
    #2
    Yes, you are right.


    Example:

    
User-agent: *
Disallow: /subfolder/
Allowed: /subfolder/subfolder2/
Disallow: /subfolder/subfolder2/sub3/
    Code (markup):
    :) Crawler can crawl your allowed directories but can't crawl disallow directories.
     
    dir18, Jul 12, 2011 IP
  3. i_am_dhaval

    i_am_dhaval Well-Known Member

    Messages:
    1,364
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    138
    #3
    User-agent: *
    Disallow: /subfolder/

    but i am not adding any tag for Allowed but still it will crawl my site right only subfolder data not come. just conforming.
     
    i_am_dhaval, Jul 12, 2011 IP
  4. dir18

    dir18 Active Member

    Messages:
    256
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    65
    #4
    You are right, crawler can crawl your all other folders and files.
     
    dir18, Jul 12, 2011 IP
  5. mrupp2006

    mrupp2006 Well-Known Member

    Messages:
    176
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    113
    #5
    dir18 is correct, Crawler will crawl everything except what is in the disallow command.
     
    mrupp2006, Jul 24, 2011 IP
  6. Abhishek Kundu

    Abhishek Kundu Active Member

    Messages:
    133
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    58
    #6
    hi dhaval,

    you are right. that code that you have written in perfectly all right to block the directories & files inside the /subfolder/.

    this will prevent all the search engine robots from indexing it.
     
    Last edited: Jul 25, 2011
    Abhishek Kundu, Jul 25, 2011 IP
  7. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #7
    This is perfect fit for your requirements.
     
    manish.chauhan, Aug 1, 2011 IP