What should a robots.txt say?

Discussion in 'robots.txt' started by Seobiznezz, Nov 7, 2007.

0
  1. #1
    What exactly should a robots.txt file say if you want your whole site spidered 24hours a day 365 days a year?
     
    Seobiznezz, Nov 7, 2007 IP
  2. hans

    hans Well-Known Member

    Messages:
    2,923
    Likes Received:
    126
    Best Answers:
    1
    Trophy Points:
    173
    #2
    then you have an EMPTY robots.txt

    because in robots.txt you EXCLUDE folders or DENY access to particular or all bots.

    if you love to be indexed by all
    you may add the new LINE

    Sitemap: /sitemapindex.xml

    STUDY (Google) what belongs into that file Sitemap: /sitemapindex.xml and in what format.

    its used to announce to ALL SE's where your
    site map files
    and
    RSS feeds can be found
     
    hans, Nov 7, 2007 IP
  3. xhenxhe

    xhenxhe Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Here is some good info on what to put in you robots file: robotstxt.org
     
    xhenxhe, Nov 9, 2007 IP
  4. jasmine1111

    jasmine1111 Peon

    Messages:
    34
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    In robots.txt file you can exclude your folders that you want that crawler should not crawl the site

    It is a good parctice to add the robot.txt file in you site so that the bandwidth of the site will not waste
     
    jasmine1111, Nov 16, 2007 IP
  5. silence

    silence Peon

    Messages:
    224
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Wow!
    Thanks for the tip guys, i didn't know about this yet haha i'm a newbie :p
     
    silence, Nov 17, 2007 IP