Robots.txt Disallow: / function

Discussion in 'robots.txt' started by aniket123, Nov 1, 2011.

0
  1. #1
    Hello Everybody,

    Can you please explain that what is the function of mentioned robots.txt.

    User-Agent: *
    Disallow: /dogs-for-sale/?*
    Disallow: /dogs-for-sale?*

    --
    Thanks
     
    aniket123, Nov 1, 2011 IP
  2. pr0t0n

    pr0t0n Well-Known Member

    Messages:
    243
    Likes Received:
    10
    Best Answers:
    10
    Trophy Points:
    128
    #2
    User-Agent: {the robot the following rule applies to}
    Disallow: {the pages you want to block}

    User-Agent: * - this line tells that the following rule applies to all search engine robots (* is used as asterisk)
    Disallow: /dogs-for-sale/ - this line would be the rule which applies to above stated robots, and it tells them NOT to index subfolder dogs-for-sale as well as anything in it. (so the asterisk is not needed there, and I'm not aware that question mark has some role in robots.txt at all)
     
    Last edited: Nov 1, 2011
    pr0t0n, Nov 1, 2011 IP
  3. aruninfo

    aruninfo Peon

    Messages:
    85
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3

    Above given statement is true.
     
    aruninfo, Nov 3, 2011 IP
  4. tripbuilder

    tripbuilder Peon

    Messages:
    72
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    that mean search engines will not follow your links and if you remove / then it will follow indexing.
     
    tripbuilder, Nov 14, 2011 IP