WGET CSV File Behind A Secure Website.

I am trying to get a CSV to automatically download from a remote site every day at a specific time. The only thing is, I need to be logged in to get that CSV.
Is there a way I can crate a script to login in, get the CSV and be on my way?

I have all the credentials to log in.

Thank You,
-Tim

 

Is this an ftp type login, or just a website authentication form?

 

Its a CMS where I can login and download a file manually. Through just clicking on a link.

I want to avoid having to login and download that CSV through a link and automate it.

No FTP involved...just in eed to enter a username/password to enter the CMS

Hope that helps..

 

you can try using curl in php.

 

Rather than just saying... examples are nice

For normal stuff like file_get_contents equivalent there is this replacement function (especially since some host disabled file_get_contents and fopen remotely) that uses cURL.

function file_get_contents_curl($url) {
	$ch = curl_init();
	
	curl_setopt($ch, CURLOPT_HEADER, 0);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Set curl to return the data instead of printing it to the browser.
	curl_setopt($ch, CURLOPT_URL, $url);
	
	$data = curl_exec($ch);
	curl_close($ch);
	
	return $data;
}

PHP:

But for SSL/HTTPS there is this (googled for it):

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $base_url);

// Set your login and password for authentication
curl_setopt($ch, CURLOPT_USERPWD, 'login:pasword');

/* 
You can use CURLAUTH_BASIC, CURLAUTH_DIGEST, CURLAUTH_GSSNEGOTIATE,
 CURLAUTH_NTLM, CURLAUTH_ANY, and CURLAUTH_ANYSAFE

 You can use the bitwise | (or) operator to combine more than one method.
 If you do this, CURL will poll the server to see what methods it supports and pick the best one.

 CURLAUTH_ANY is an alias for CURLAUTH_BASIC | CURLAUTH_DIGEST |
 CURLAUTH_GSSNEGOTIATE | CURLAUTH_NTLM

 CURLAUTH_ANYSAFE is an alias for CURLAUTH_DIGEST | CURLAUTH_GSSNEGOTIATE |
 CURLAUTH_NTLM

 Personally I prefer CURLAUTH_ANY as it covers all bases
*/
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);

/* This is occassionally required to stop CURL from verifying the peer's certificate.
 CURLOPT_SSL_VERIFYHOST may also need to be TRUE or FALSE if
 CURLOPT_SSL_VERIFYPEER is disabled (it defaults to 2 - check the existence of a
 common name and also verify that it matches the hostname provided)
*/

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

// Optional: Return the result instead of printing it
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

// The usual - get the data and close the session
$data = curl_exec($ch);
curl_close($ch);

PHP:

And course the return is in $data

 

Well do this using curl http://blog.taragana.com/index.php/...php-for-authentication-and-ssl-communication/

 

Look above.

 

oh well i guess i was seeing the code and posting and u had just got it .. cool..

 

Gota love google.

 

Wow thanks for all the feedback and help. This is what I am testing right now...
I can successfully get the data.. the only thing is,..its just giving me the data in the page, it is not downloading the actual csv.

this is the code:

function curl_login($url,$data,$proxy,$proxystatus){
$fp = fopen("cookie.txt", "w");
fclose($fp);
$login = curl_init();
curl_setopt($login, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($login, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($login, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($login, CURLOPT_TIMEOUT, 40);
curl_setopt($login, CURLOPT_RETURNTRANSFER, TRUE);
if ($proxystatus == 'on') {
curl_setopt($login, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($login, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($login, CURLOPT_PROXY, $proxy);
}
curl_setopt($login, CURLOPT_URL, $url);
curl_setopt($login, CURLOPT_HEADER, TRUE);
curl_setopt($login, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($login, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($login, CURLOPT_POST, TRUE);
curl_setopt($login, CURLOPT_POSTFIELDS, $data);
ob_start(); // prevent any output
return curl_exec ($login); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($login);
unset($login);
}

function curl_grab_page($site,$proxy,$proxystatus){
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
if ($proxystatus == 'on') {
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
ob_start(); // prevent any output
return curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($ch);
}



curl_login('HIDDEN','','off');

curl_grab_page('HIDDEN','','off');

 

Any ideas?... im so close i can feel it!!

 

function curl_grab_page($site,$proxy,$proxystatus){
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
if ($proxystatus == 'on') {
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
ob_start(); // prevent any output
return curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($ch);
} 

PHP:

First off, you're returning so the last two lines will never execute.. Also don't know why you'd need ob_start etc..

Replace:

ob_start(); // prevent any output
return curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($ch);

PHP:

with:

$output = curl_exec ($ch); // execute the curl command
curl_close ($ch);
return $output;

PHP:

$output = curl_grab_page('HIDDEN','','off'); 

PHP:

Then just output it or save it to a page:

file_put_contents('filename.txt', $output); 

PHP:

 

Oh, and if you want it to pop up a download box in the browser, you can do this:

$size = strlen($output);
$filename = 'download.txt';

header('Pragma: public');
header('Expires: 0');
header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
header('Content-Type: application/octet-stream');
header('Content-Transfer-Encoding: binary');
header('Content-Length: ' . $size);
header('Content-Disposition: attachment; filename="' . $filename . '"');

echo $output;
die();

PHP:

 

You rock! Thank you. but what if i want it to save it to a folder automatically? without popping up the download box? +rep

 

Thanks

Re: On your server? then just use file_put_contents.

 

hmm It is creating the file but it is 0kb...getting there.. any initial thoughts? thank you very much for your help so far..

 

It was caching on my end... your help worked!! thank you!!!!!!