Can anyone find any mistakes in my robots.txt

Discussion in 'robots.txt' started by MediaHustler, Mar 24, 2006.

0
  1. #1
    I really don't wanna accidentally block anything I don't want blocked, can someone review my robots.txt

    http://www.DumbNerd.com/robots.txt

    I want it to not crawl
    /media/
    /links2/
    /comments/
    /includes/

    But I wan't it to crawl everything else. Did I get it right?
     
    MediaHustler, Mar 24, 2006 IP
  2. nicknick

    nicknick Peon

    Messages:
    429
    Likes Received:
    21
    Best Answers:
    0
    Trophy Points:
    0
    #2
    looks good to me.
     
    nicknick, Mar 24, 2006 IP
  3. MediaHustler

    MediaHustler Well-Known Member

    Messages:
    1,857
    Likes Received:
    46
    Best Answers:
    0
    Trophy Points:
    120
    #3
    Thanks, i've used robots.txt alot but never actually blocked anything :D
     
    MediaHustler, Mar 24, 2006 IP
  4. alifan

    alifan Peon

    Messages:
    46
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    fine no probs with that
     
    alifan, Mar 30, 2006 IP