can you use wget to spider a site ?

Discussion in 'Programming' started by doridori, Nov 16, 2008.

0
  1. #1
    can you use wget to spider a site ?

    wget -mirror or something like that to first downloda the majority of the site...html, and texts, to the hard drive, then scraping the results? this way i wont cause excessive load on my server and the target website........
     
    doridori, Nov 16, 2008 IP
  2. NuLLByTe

    NuLLByTe Active Member

    Messages:
    382
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    80
    #2
    wget has an argument, -spider.
     
    NuLLByTe, Nov 16, 2008 IP