1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Program to get all urls in a site?

Discussion in 'General Chat' started by mines, Oct 24, 2007.

0
  1. #1
    Hi
    is there any product/site/tool that can crawl a site and return a plain list of all urls its crawled? I've written a basic one in php but its too slow.

    thanks
     
    mines, Oct 24, 2007 IP
  2. mightyb

    mightyb Banned

    Messages:
    6,566
    Likes Received:
    405
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Yep... It was designed for sitemap creation. You can define the depth of the crawl and so on. I can't remember what its called though. Search for sitemap spider generator or something along the lines.
     
    mightyb, Oct 24, 2007 IP
  3. mines

    mines Well-Known Member

    Messages:
    1,127
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    160
    #3
    Oh yeh. i tried http://www.sitemapdoc.com/ but its limited to 500. (im trying to crawl several thousand). Ill try some other sitemap tools though. :)
     
    mines, Oct 24, 2007 IP
  4. tarponkeith

    tarponkeith Well-Known Member

    Messages:
    4,758
    Likes Received:
    279
    Best Answers:
    0
    Trophy Points:
    180
    #4
    tarponkeith, Oct 24, 2007 IP
  5. mightyb

    mightyb Banned

    Messages:
    6,566
    Likes Received:
    405
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thats not it. The program i used used to give me around 50k links. But yea, close enough :D
     
    mightyb, Oct 24, 2007 IP
  6. Monty

    Monty Peon

    Messages:
    1,363
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Monty, Oct 24, 2007 IP