Using wget to download files in a directory, that are named 1-4000

Discussion in 'Site & Server Administration' started by shacow, May 20, 2008.

  1. #1
    This is really confusing me.

    I can wget -r ftp to download all files in a folder, but because of a server truncating at 2000 records its not possible to download all of the 4000 files.

    I can access the files via wget http. although there is no folder list that gives me all the file names, however, I know that the files are named, 1 - 4000.txt.

    how would I download all of the files, starting at 1 and ending at 4000, when all the extensions are the same.

    would this be a bash script that generates a list of the files?

    someone please help me as i need to move all my files from my old server onto my new server before my old server gets closed down.


    warm regards, Shaun
     
    shacow, May 20, 2008 IP
  2. shacow

    shacow Active Member

    Messages:
    339
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    60
    #2
    I need to generate a list of all of the file locations 1-4000 and then use wget to download each one after the previous one has completed.

    could anyone tell me how to do this?

    thanks, Shaun
     
    shacow, May 20, 2008 IP
  3. krakjoe

    krakjoe Well-Known Member

    Messages:
    1,795
    Likes Received:
    141
    Best Answers:
    0
    Trophy Points:
    135
    #3
    
    #!/bin/bash
    for i in `seq 1 4000`;
    do
    	echo "Gonna fetch http://url.com/path/$i.txt"
    	wget http://url.com/path/$i.txt 2>1 1>/dev/null
    done
    Code (markup):
    replace http://url.com/path/$i.txt with the approprite url ...
     
    krakjoe, May 20, 2008 IP
  4. shacow

    shacow Active Member

    Messages:
    339
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    60
    #4
    Thank you! you are a life saver! I owe you a beer!
     
    shacow, May 20, 2008 IP
  5. agnivo007

    agnivo007 Peon

    Messages:
    4,290
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Ouch, I had thought of the same shell script only to find it has been done while replying! :p

    Nice it worked!
     
    agnivo007, May 20, 2008 IP