so? robot.txt?

Discussion in 'Search Engine Optimization' started by kidblogger, May 1, 2007.

0
  1. #1
    Ok, I have made my robot.txt.. but how do i use it? could you guyz give me guide on how to use it properly..
     
    kidblogger, May 1, 2007 IP
  2. kidblogger

    kidblogger Active Member

    Messages:
    791
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    70
    #2
    anyone knows?
     
    kidblogger, May 1, 2007 IP
  3. dougadam

    dougadam Active Member

    Messages:
    495
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    58
    #3
    The robots.txt file is not intended for controlling access, it main purpose is to disallow access to certain parts of your site and only if the robot visiting obeys the Robots Exclusion Protocol.
     
    dougadam, May 1, 2007 IP
  4. d16man

    d16man Well-Known Member

    Messages:
    6,900
    Likes Received:
    160
    Best Answers:
    0
    Trophy Points:
    180
    #4
    get rid of it and use meta tags instead...robots.txt is an old school way of doing it, imo anyway.
     
    d16man, May 1, 2007 IP
  5. boyponga

    boyponga Banned

    Messages:
    1,013
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Upload it to your FTP server and the robots.txt will do the rest
     
    boyponga, May 1, 2007 IP
  6. Linkmeup

    Linkmeup Peon

    Messages:
    197
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Once it's uploaded to your site you don't have to do anything with it.
     
    Linkmeup, May 1, 2007 IP
  7. kidblogger

    kidblogger Active Member

    Messages:
    791
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    70
    #7
    Ow.. Thanks :D
     
    kidblogger, May 2, 2007 IP
  8. Dudibob

    Dudibob Peon

    Messages:
    618
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #8
    careful with it though, it's made to ban the search engines from areas of your site, or even your whole site! I use a blank one so the search engines get a 200 response when they get it but there's nothing in it :)
     
    Dudibob, May 2, 2007 IP
  9. seoperson

    seoperson Peon

    Messages:
    501
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #9
    this is an example robots.txt file.
    # All robots will spider the domain
    User-agent: *
    #below line means that everything is accessible to each bot
    Disallow:
    #if you do not want to allow access to particular directory,then put forward slash and dir name after Disallow:
     
    seoperson, May 2, 2007 IP
  10. kidblogger

    kidblogger Active Member

    Messages:
    791
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    70
    #10
    ow thanks for informing :)
     
    kidblogger, May 5, 2007 IP