What this robot.txt will do ??

Discussion in 'robots.txt' started by Rajnish357, Mar 26, 2012.

0
  1. #1
    Please Let me know what this content will do in the robot.txt

    
User-agent: *

Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?replytocom= 
Disallow: /category/
Disallow: /tag/
    Code (markup):

     
    Rajnish357, Mar 26, 2012 IP
  2. Econox

    Econox Well-Known Member

    Messages:
    478
    Likes Received:
    13
    Best Answers:
    2
    Trophy Points:
    140
    #2
    Web crawlers won't be able to access those directories.
     
    Econox, Mar 26, 2012 IP
  3. xoomcc

    xoomcc Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Registrictions for Web crawlers like google or yahoo
     
    xoomcc, Mar 27, 2012 IP
  4. anna30

    anna30 Well-Known Member

    Messages:
    280
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    123
    #4
    Web crawler will not crawl the mentioned five directory and files under those directory.
     
    anna30, Apr 3, 2012 IP
  5. henrywilliams

    henrywilliams Peon

    Messages:
    119
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Crawler won't be able to see and crawl these directories.
     
    henrywilliams, Apr 6, 2012 IP
  6. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    just means those areas wont be placed on search engines to be seen.
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  7. chitra19

    chitra19 Peon

    Messages:
    28
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    This robots.txt will allow all the search engines to crawl over your site but the mentioned directories alongwith all files and folders in these directories will not be crawled by the search engines.
     
    chitra19, Aug 13, 2012 IP
  8. ChR0n1k

    ChR0n1k Well-Known Member

    Messages:
    649
    Likes Received:
    54
    Best Answers:
    0
    Trophy Points:
    170
    #8
    This guy literally just replied, "crawler that is crawling" NOT only does that not make any sense, I feel like I am going crazy that this spammer isn't banned yet. This entire branch of the forum seems like a post count bumper.
     
    ChR0n1k, Aug 19, 2012 IP
  9. rjcipa

    rjcipa Greenhorn

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #9
    google crawler will not crawl the mentioned five directory.
     
    rjcipa, Aug 21, 2012 IP