How does robots.txt help?

Discussion in 'robots.txt' started by jonyroufe, Nov 8, 2009.

0
  1. #1
    I'm building a new website, can anyone tell me how does robots.txt help?
     
    jonyroufe, Nov 8, 2009 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    Robots.txt is used to instruct bots what parts of your website, you want to show them and what not. Suppose you have a private folder that you do not want to be indexed then you can simply use:

    user-agent: *
    Disallow: /private
     
    manish.chauhan, Nov 9, 2009 IP
  3. Bohra

    Bohra Prominent Member

    Messages:
    12,573
    Likes Received:
    537
    Best Answers:
    0
    Trophy Points:
    310
    #3
    you can just control the stuff u dont want to make public or if u want to block any particular search engine
     
    Bohra, Nov 10, 2009 IP
  4. helplinelaw

    helplinelaw Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You are created a robots.txt file of a website....
    that used a nofollow any webpages than add a that file in robots.txt
     
    helplinelaw, Nov 11, 2009 IP