1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt

Discussion in 'robots.txt' started by harryoxford, Feb 10, 2011.

0
  1. #1
    Is this necessary to add robots.txt file in my site.
     
    harryoxford, Feb 10, 2011 IP
  2. BlackFriday

    BlackFriday Active Member

    Messages:
    741
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    95
    #2
    I think you should

    Robots.txt is a plain text file and uploading to the root directory of your site.
    Spiders seek first this file and processed prior to index your site.Robots.txt tells spiders which pages to crawl.

    example:
    1.
    User-agent: *
    Disallow:

    2.
    User-agent: *
    Disallow: /faq/

    3.
    User-agent: *
    Disallow: /faq/
    Disallow: /info/about/

    We prohibit the directories and even individual spiders to crawl our site.
    I hope I was helpful
     
    BlackFriday, Feb 10, 2011 IP
  3. GuidoB

    GuidoB Member

    Messages:
    327
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    38
    #3
    Yes it is very usefull for your SEO!
     
    GuidoB, Feb 12, 2011 IP
  4. gred

    gred Member

    Messages:
    30
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    43
    #4
    it is usefull but not necessary
     
    gred, Feb 14, 2011 IP
  5. ACME Squares

    ACME Squares Peon

    Messages:
    98
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Having one stops the error log filling up with 404s.
    Robots.txt is also the best way to announce your sitemap to all search engines at once.
     
    ACME Squares, Feb 24, 2011 IP
  6. stock_stock

    stock_stock Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    can u tell me how robot.txt can stop 404 error
     
    stock_stock, Mar 3, 2011 IP
  7. alsgroup

    alsgroup Member

    Messages:
    148
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    48
    #7
    Generally robots.txt file use to restrict Google's crawler to not cached your unneccsary files or folder from the server.
     
    alsgroup, Mar 8, 2011 IP
  8. Sujeet Yadav

    Sujeet Yadav Peon

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    it is not necessary
     
    Sujeet Yadav, Mar 9, 2011 IP
  9. Blackcastle

    Blackcastle Member

    Messages:
    107
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #9
    Yes it is necessary!

    Check out wikipedia and facebooks robots.txt files:-

    http://facebook.com/robots.txt
    http://en.wikipedia.org/robots.txt

    They can be as detailed as these or as simple as this:-

    User-agent: *
    Allow: /

    If you don't know how to do it you can just go to Google Webmaster Tools where you can easily generate a robots.txt file for your site.
     
    Blackcastle, Mar 13, 2011 IP
  10. Insight

    Insight Peon

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Yes, I definitely agree with GuidoB, because it plays a part in SEO.
     
    Insight, Mar 15, 2011 IP
  11. beven

    beven Well-Known Member

    Messages:
    483
    Likes Received:
    1
    Best Answers:
    1
    Trophy Points:
    113
    #11
    I have no idea to do develop this file or is it copy paste......Plz give me explanation
     
    beven, Mar 17, 2011 IP
  12. Ivan.Muttaqin

    Ivan.Muttaqin Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    bingung,,,,
    :( :( :(
     
    Ivan.Muttaqin, Mar 19, 2011 IP
  13. Ilona Andrews

    Ilona Andrews Peon

    Messages:
    295
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    hi, Seo point of view, robot.txt necessary .
     
    Ilona Andrews, Apr 18, 2011 IP
  14. zeninho

    zeninho Active Member

    Messages:
    129
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #14
    yes it is!! VERY MUCH !!! :)
     
    zeninho, Apr 18, 2011 IP
  15. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #15
    If you don't have a robots.txt file, your web server will return a 404 error page to the engine instead. For those who have customized their 404 error document, that customised 404 page will end up being sent to the spider repeatedly throughout the day. Now, if you have customized your 404 page, chances are that it's bigger than the standard server error message "404 File Not Found" (since you will want your error page to say more than the default error message). In other words, failing to create a robots.txt will cause the search engine spider to use up more of your bandwidth as a result of its repeated retrieval of your large 404 error file. (How much more depends, of course, on the size of your 404 error page.)
     
    manish.chauhan, May 4, 2011 IP