Program to get all urls in a site?

Discussion in 'General Chat' started by mines, Oct 24, 2007.

  1. #1
    Hi
    is there any product/site/tool that can crawl a site and return a plain list of all urls its crawled? I've written a basic one in php but its too slow.

    thanks
     
    mines, Oct 24, 2007 IP
  2. mightyb

    mightyb Banned

    Messages:
    6,566
    Likes Received:
    405
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Yep... It was designed for sitemap creation. You can define the depth of the crawl and so on. I can't remember what its called though. Search for sitemap spider generator or something along the lines.
     
    mightyb, Oct 24, 2007 IP
  3. mines

    mines Well-Known Member

    Messages:
    1,127
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    160
    #3
    Oh yeh. i tried http://www.sitemapdoc.com/ but its limited to 500. (im trying to crawl several thousand). Ill try some other sitemap tools though. :)
     
    mines, Oct 24, 2007 IP
  4. tarponkeith

    tarponkeith Well-Known Member

    Messages:
    4,758
    Likes Received:
    279
    Best Answers:
    0
    Trophy Points:
    180
  5. mightyb

    mightyb Banned

    Messages:
    6,566
    Likes Received:
    405
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thats not it. The program i used used to give me around 50k links. But yea, close enough :D
     
    mightyb, Oct 24, 2007 IP
  6. Monty

    Monty Peon

    Messages:
    1,363
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Monty, Oct 24, 2007 IP