1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Some help please...

Discussion in 'robots.txt' started by cgo85, Dec 28, 2005.

0
  1. #1
    Can someone tell if this is a valid robots.txt:

    User-agent: Teoma
    Disallow: /
    User-agent: psbot
    Disallow: /
    User-agent: appie
    Disallow: /
    User-agent: Accoona-AI-Agent/1.1.1
    Disallow: /
    User-agent: askjeeves
    Disallow: /


    This doesn't seem to be working. Your help is appreciated.
     
    cgo85, Dec 28, 2005 IP
  2. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #2
    What do you mean when you say "this doesn't seem to be working"?

    What that file is doing is instructing a limited number of spiders (user agents)to NOT index the entire site. All remaining spiders, by default, are given permission to index the entire site. Is this what you intended to do?

    Usually, you would leave a blank line between successive user-agent instructions but I must admit I don't know if spiders carfe or if this is usually done to improve readability for humans.
     
    minstrel, Dec 29, 2005 IP
  3. Integrov

    Integrov Guest

    Messages:
    79
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    i thinks that means that you are restricting each of the search engines you had listed...

    if you wanted to do so, then the standard is to put a newline between each disalow...
     
    Integrov, Jan 1, 2006 IP