1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Benefits and use of robots.txt file

Discussion in 'robots.txt' started by umairraja52, Oct 29, 2009.

0
  1. #1
    I wnat to know what are the benefits and use of robots.txt file is. How can i use this file in my website.

    Is it necessary or not for my website future. :confused:
     
    umairraja52, Oct 29, 2009 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    Robots.txt file is basically used to instruct crawlers what files/folders to crawl or not. If you have some private information on your website that you do not want to share with others, you can simply block that file/folder using robots.txt.

    You can simply create a robots.txt file and upload that in the root folder. you can set the instruction as required in your website.
     
    manish.chauhan, Oct 29, 2009 IP
  3. boyle7

    boyle7 Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    thanks for the sharing known
     
    boyle7, Nov 4, 2009 IP
  4. php.freak

    php.freak Well-Known Member

    Messages:
    185
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #4
    using this file you can block you files and folders ..so that crawler can not read them.....
     
    php.freak, Nov 11, 2009 IP
  5. mjreine

    mjreine Well-Known Member

    Messages:
    110
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #5
    Let an app generate this file for you then upload it to the root of your site...simple
    search google for robots.txt generator
     
    mjreine, Nov 15, 2009 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    I would recommend you to generate the robots.txt yourself as no application could configure what files or directories to be allowed and viceversa.
     
    manish.chauhan, Nov 17, 2009 IP