robots

Discussion in 'robots.txt' started by noelves, Nov 28, 2005.

0
  1. #1
    I realize link popularity is kicking meta tags to the curb, but I include them anyway.

    When I submitted to "Scrub The Web" they sent a message saying my site may not be search engine friendly because my robot.txt tags are either missing or configured wrong.

    This is the tags I use: <META NAME="robots" CONTENT="FOLLOW,INDEX">

    It went on the say I needed a robot.txt file in my root folder. I done some research on the web concerning robot meta tags, and only a couple of the articles mentioned using a robot file. The rest of them suggests using the tags like I have above. Which one is right?

    Thanks,
    noelves
     
    noelves, Nov 28, 2005 IP
  2. frankm

    frankm Active Member

    Messages:
    915
    Likes Received:
    63
    Best Answers:
    0
    Trophy Points:
    83
    #2
    robots.txt is site wide, the <meta name="Robots" ...> is per page.
    so both are right I guess.

    check out www.robotstxt.org
     
    frankm, Nov 28, 2005 IP
  3. noelves

    noelves Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    OH! OK!

    I'm doing per page, so I should be ok. Thanks for the help!

    noelves
     
    noelves, Nov 28, 2005 IP