External URLS

Discussion in 'robots.txt' started by lordmenace, Sep 20, 2006.

0
  1. #1
    What would I need to add to my robots.txt file to exclude external urls?
     
    lordmenace, Sep 20, 2006 IP
  2. bentong

    bentong Banned

    Messages:
    3,543
    Likes Received:
    257
    Best Answers:
    0
    Trophy Points:
    0
    #2
    if you mean you dont want the bot to follow the link, just add
    on href...
     
    bentong, Sep 22, 2006 IP
  3. lordmenace

    lordmenace Well-Known Member

    Messages:
    541
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    140
    #3
    I have MANY outbound links on my site and I can't add teh nofollow tag to them all. It would take forever. So I figured there might be a way to do it with robots.txt
     
    lordmenace, Sep 24, 2006 IP
  4. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #4
    robots.txt is for internal URL's only.

    Jean-Luc
     
    Jean-Luc, Sep 24, 2006 IP
  5. rosiee007

    rosiee007 Notable Member

    Messages:
    3,352
    Likes Received:
    179
    Best Answers:
    0
    Trophy Points:
    230
    #5
    Do you have a dynamic site, or have you given static links from within your website pages to these websites? Are you taking about the links on your link pages or from the web pages?
     
    rosiee007, Sep 25, 2006 IP
  6. lordmenace

    lordmenace Well-Known Member

    Messages:
    541
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    140
    #6
    They are static links.
     
    lordmenace, Oct 7, 2006 IP
  7. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #7
    add
    disallow : /the_external_url_that_you_dont_to_be_indexed.html

    if you add this line to your robots.txt , your problem will be solved
     
    trichnosis, Dec 18, 2006 IP