1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Advantages of robots.txt

Discussion in 'robots.txt' started by areaba, Apr 25, 2009.

  1. #1
    Friends i am new in seo field , I have just read about robots.txt file, but i am so confuse about this. Can anyone tell me what is the robots.txt file, why we need this , what the advantages of this in seo. whats things we mention in that file and how we can create this.
    SEMrush
    Thanks
     
    areaba, Apr 25, 2009 IP
    SEMrush
  2. pipes

    pipes Prominent Member

    Messages:
    12,767
    Likes Received:
    958
    Best Answers:
    0
    Trophy Points:
    360
    #2
    My knowledge of the robots.txt is minimal but i know that you can block certain urls from being indexed by the search engines.
     
    pipes, Apr 26, 2009 IP
  3. Lovely

    Lovely Well-Known Member

    Messages:
    2,997
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    155
    #3
    Robots.txt won't have a direct effect in site promotion. It is a plain text file that sits in the root directory of your site and tells various search engine spiders that they shouldn't index all or certain parts of your site.
    Robots.txt is beneficial if you have parts of your website that are very similar you can block them from being crawled to avoid being flagged as a spammer.
     
    Lovely, Apr 28, 2009 IP
  4. blackworld

    blackworld Peon

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    yes robots.txt help us preventing private urls from indexing by various search engines which we donot want i know this but i have to find how to code robots.txt
     
    blackworld, Apr 28, 2009 IP
  5. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #5
    robots.txt can be used for blocking duplicate content in some cases

    Cheers,
     
    Lpe04, Apr 28, 2009 IP
  6. toycat

    toycat Well-Known Member

    Messages:
    2,304
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    100
    #6
    ANy info on HOW you block parts of your site?
     
    toycat, May 1, 2009 IP
  7. sem-profiteer

    sem-profiteer Active Member

    Messages:
    257
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    75
    #7
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /stats/
     
    sem-profiteer, May 7, 2009 IP
  8. faiz.khan

    faiz.khan Active Member

    Messages:
    46
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    76
    #8
    I think Lovely has explain it all !

    I would just like to add few things in that

    Robots.txt is nothing but just a simple instructions file for search engine crawlers which guide them which page need to crawl and which to skip.

    - for security reason if you need to block few pages to be crawled by any search engine crawler then robots.txt is the best option to do it.

    Thanks
    Faiz Khan


     
    faiz.khan, May 11, 2009 IP
  9. alus1onZ

    alus1onZ Member

    Messages:
    96
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #9
    Google "robots.txt" and you will find TONS of information and guides how to use it and what advantages have this file.
     
    alus1onZ, May 11, 2009 IP
  10. dotcompals

    dotcompals Prominent Member

    Messages:
    2,905
    Likes Received:
    254
    Best Answers:
    0
    Trophy Points:
    320
    #10
    dotcompals, May 11, 2009 IP