1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

is am i right with robots.txt file

Discussion in 'robots.txt' started by i_am_dhaval, Jul 11, 2011.

  1. #1
    One of my site i am using this
    
    User-agent: *
    Disallow: /subfolder/
    Code (markup):
    is this mean all pages are allow from the site only pages which is in " subfolder " that is not allow?

    am i right?
     
    i_am_dhaval, Jul 11, 2011 IP
  2. dir18

    dir18 Active Member

    Messages:
    256
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    65
    #2
    Yes, you are right.


    Example:

    
    User-agent: *
    Disallow: /subfolder/
    Allowed: /subfolder/subfolder2/
    Disallow: /subfolder/subfolder2/sub3/
    
    Code (markup):
    :) Crawler can crawl your allowed directories but can't crawl disallow directories.
     
    dir18, Jul 12, 2011 IP
  3. i_am_dhaval

    i_am_dhaval Well-Known Member

    Messages:
    1,364
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    138
    #3
    User-agent: *
    Disallow: /subfolder/

    but i am not adding any tag for Allowed but still it will crawl my site right only subfolder data not come. just conforming.
     
    i_am_dhaval, Jul 12, 2011 IP
  4. dir18

    dir18 Active Member

    Messages:
    256
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    65
    #4
    You are right, crawler can crawl your all other folders and files.
     
    dir18, Jul 12, 2011 IP
  5. mrupp2006

    mrupp2006 Well-Known Member

    Messages:
    176
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    113
    #5
    dir18 is correct, Crawler will crawl everything except what is in the disallow command.
     
    mrupp2006, Jul 24, 2011 IP
  6. Abhishek Kundu

    Abhishek Kundu Active Member

    Messages:
    133
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    58
    #6
    hi dhaval,

    you are right. that code that you have written in perfectly all right to block the directories & files inside the /subfolder/.

    this will prevent all the search engine robots from indexing it.
     
    Last edited: Jul 25, 2011
    Abhishek Kundu, Jul 25, 2011 IP
  7. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #7
    This is perfect fit for your requirements.
     
    manish.chauhan, Aug 1, 2011 IP