1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt Disallow: / function

Discussion in 'robots.txt' started by aniket123, Nov 1, 2011.

0
  1. #1
    Hello Everybody,

    Can you please explain that what is the function of mentioned robots.txt.

    User-Agent: *
    Disallow: /dogs-for-sale/?*
    Disallow: /dogs-for-sale?*

    --
    Thanks
     
    aniket123, Nov 1, 2011 IP
  2. pr0t0n

    pr0t0n Well-Known Member

    Messages:
    243
    Likes Received:
    10
    Best Answers:
    10
    Trophy Points:
    128
    #2
    User-Agent: {the robot the following rule applies to}
    Disallow: {the pages you want to block}

    User-Agent: * - this line tells that the following rule applies to all search engine robots (* is used as asterisk)
    Disallow: /dogs-for-sale/ - this line would be the rule which applies to above stated robots, and it tells them NOT to index subfolder dogs-for-sale as well as anything in it. (so the asterisk is not needed there, and I'm not aware that question mark has some role in robots.txt at all)
     
    Last edited: Nov 1, 2011
    pr0t0n, Nov 1, 2011 IP
  3. aruninfo

    aruninfo Peon

    Messages:
    85
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3

    Above given statement is true.
     
    aruninfo, Nov 3, 2011 IP
  4. tripbuilder

    tripbuilder Peon

    Messages:
    72
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    that mean search engines will not follow your links and if you remove / then it will follow indexing.
     
    tripbuilder, Nov 14, 2011 IP