robot.txt

Discussion in 'General Chat' started by Wrongfully Convicted, Nov 7, 2008.

0
  1. #1
    Where do i get a robot.txt.from and what do i do with it once i get it?
     
    Wrongfully Convicted, Nov 7, 2008 IP
  2. GotWebHost

    GotWebHost Peon

    Messages:
    353
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You create the file yourself and you use it to tell search engine spiders to stay away from certain pages on your site.
     
    GotWebHost, Nov 7, 2008 IP
  3. !Unreal

    !Unreal Well-Known Member

    Messages:
    1,671
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    165
    #3
    !Unreal, Nov 7, 2008 IP
  4. nursekatie04

    nursekatie04 Guest

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    you may want to look into doing something like this with .htaccess since many bots ignore the robots.txt
     
    nursekatie04, Nov 7, 2008 IP