LF:Software to get all links on a site?

Discussion in 'Link Development' started by wheel, Apr 30, 2006.

  1. #1
    I'm looking for software (free or paid) that will pull all the outbound links from a site. Basically I want to input a url, and end up with a list of all the outbound URLS on any page of the site. I don't care about checking validity, I just want a list.

    Anything out there do this? Lots of link checkers and stuff, but it's never clear if the perform the function that I need.

    TIA.
     
    wheel, Apr 30, 2006 IP
  2. exam

    exam Peon

    Messages:
    2,434
    Likes Received:
    120
    Best Answers:
    0
    Trophy Points:
    0
    #2
    exam, Apr 30, 2006 IP
  3. BILZ

    BILZ Peon

    Messages:
    1,515
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Xenu's link sleuth is typically used to help you find borken links, but i think it will also tell you what the outbound links are also. I havent used it in a while. Give it try see if it works: http://home.snafu.de/tilman/xenulink.html
     
    BILZ, Apr 30, 2006 IP
  4. BILZ

    BILZ Peon

    Messages:
    1,515
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hehe, we posted at the same time. Great minds think alike :)
     
    BILZ, Apr 30, 2006 IP
    exam likes this.
  5. wheel

    wheel Peon

    Messages:
    477
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thanks folks!

    If I run this on someone else's site, does it obey robots.txt and behave politely?
     
    wheel, Apr 30, 2006 IP
  6. BILZ

    BILZ Peon

    Messages:
    1,515
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    0
    #6
    not sure. read through the xenu site if you are concerned about it.
     
    BILZ, May 1, 2006 IP
  7. wheel

    wheel Peon

    Messages:
    477
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #7
    It doesn't obey robots.txt, so I'll check it manually.

    That's nice software, works well. Hard on a site I would think though. Defaults to 30 open connections at once. I turned it down to 1 :)

    Thanks again for the recommendations folks. Looks like exactly what I needed.
     
    wheel, May 1, 2006 IP