1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robot.txt

Discussion in 'robots.txt' started by sportsguru, Mar 16, 2005.

0
  1. #1
    I hope this is the right forum for this question. Is there some way using a robot.txt file to exclude a directory from finding my index page? Reason being I have asked this certain directory to please remove my listing and for some weird reason they won't do it?!? They say they can list whatever site they want, etc.

    I've heard you can prevent certain engines or pages from crawling your pages; but what about a directory? Can this be done? If so, I know next to nothing about robot.txt. Can someone explain how to set this up?
     
    sportsguru, Mar 16, 2005 IP
  2. Crazy_Rob

    Crazy_Rob I seen't it!

    Messages:
    13,157
    Likes Received:
    1,366
    Best Answers:
    0
    Trophy Points:
    360
    #2
    robots.txt is just directions for robots. It's not actually going to keep anyone out.

    You can use .htaccess to ban certian IP's or user_agents. But that's not going to help you in this case.
     
    Crazy_Rob, Mar 16, 2005 IP
  3. redking

    redking Member

    Messages:
    93
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    43
    #3
    If they won't change a listing you might have to take legal action.
     
    redking, Mar 16, 2005 IP
  4. fryman

    fryman Kiss my rep

    Messages:
    9,604
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    370
    #4
    Legal action? Why? What are they doing wrong?
     
    fryman, Mar 16, 2005 IP
  5. riziko

    riziko Peon

    Messages:
    189
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Can .htaccess block by referrer? That might work.
     
    riziko, Mar 16, 2005 IP