1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

What is robots.txt?

Discussion in 'Search Engine Optimization' started by misohoni, May 8, 2004.

0
  1. #1
    Should I create this file on my server and what should I put in it?

    Is it similar to the spider = index, follow tags?
     
    misohoni, May 8, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    digitalpoint, May 8, 2004 IP
  3. Cricket

    Cricket Well-Known Member

    Messages:
    182
    Likes Received:
    41
    Best Answers:
    0
    Trophy Points:
    155
    #3
    I wrote an article a while back that covers the basics of robots.txt

    This may help you a little.

    Create Robots.txt File
     
    Cricket, May 8, 2004 IP
  4. misohoni

    misohoni Notable Member

    Messages:
    1,717
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    200
    #4
    Thanks, very useful!
     
    misohoni, May 9, 2004 IP
  5. wsks

    wsks Peon

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    A great place to start read about it will be here
    www.robotstxt.org


    tells you everything you ever need to know about robot text file. :D
     
    wsks, May 16, 2004 IP
  6. aishaarora

    aishaarora Member

    Messages:
    262
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #6
    The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website.
     
    aishaarora, Nov 23, 2010 IP
  7. aaronwalter

    aaronwalter Guest

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Nice. Thanks for sharing your thoughts Cricket. Looking forward to reading more.
     
    aaronwalter, Nov 23, 2010 IP
  8. Seo Joy

    Seo Joy Active Member

    Messages:
    408
    Likes Received:
    1
    Best Answers:
    2
    Trophy Points:
    63
    #8
    Very informative link...
     
    Seo Joy, Nov 23, 2010 IP