multiplies file download with CURL by Cronjob

Discussion in 'PHP' started by trecords, Dec 18, 2011.

  1. #1
    Hi All,

    I have a database with urls to download files and save them in my server. I use cron job to download every URL one by one on every cron run. The problem is CURL not downloads second file if first file is downloading. Cron jobs runs every minute and downloader can`t finish downloading file on second cron run and also it is not downloads two or more files, and as i said waits for other download finishes.
    My Question is how can i make CURL download 1 file on every cron run no mutter if there other curl downloads file or not. My CURL code:
    	
    	$ch = curl_init();
    	$useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";
    	$fp = fopen($script_path.'/temp/'.$filename, 'w');
    	curl_setopt($ch, CURLOPT_URL, $url);
    	curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
    	curl_setopt($ch, CURLOPT_FILE, $fp);
    	curl_exec ($ch);
    	curl_close ($ch);
    	fclose($fp);
    PHP:

     
    trecords, Dec 18, 2011 IP
  2. dixcoder

    dixcoder Member

    Messages:
    71
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    26
    #2
    its obvious without downloading finish the first file it wont starts for the second because the process is serial you need to create multiple process or in another words threading to let the script do multi tasking check this url for threading in php http://semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading download the library and implement in your system. I hope this will solve your problem ;)
     
    dixcoder, Dec 19, 2011 IP
  3. sarahk

    sarahk iTamer Staff

    Messages:
    28,899
    Likes Received:
    4,555
    Best Answers:
    123
    Trophy Points:
    665
    #3
    Create multiple cron jobs to simulate threads. Have a database holding the info about the files that need downloading and when a download commences the record is updated.

    Each cron
    * selects a file
    * updates the record
    * downloads the file
    * selects the next valid file
    * updates the record
    ... and so on until all the files have a new timestamp, flag or whatever.
     
    sarahk, Dec 19, 2011 IP