Error : send parameters with php curl

Hello

i'am trying to send parameters via php curl in a stock screener to have the result in this page:
https://www.investing.com/stock-screener/?sp=country::22|sector::a|industry::a|equityType::a|exchange::9%3Eeq_market_cap;1

with the code i have the stock screner page but it's not the same at the page above: can someone help me? thank

$url ="https://www.investing.com/stock-screener/";
$parameters ="sp=country::22|sector::a|industry::a|equityType::a|exchange::9>eq_market_cap;1";

$cookie = tmpfile();
$userAgent ='Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.31';
$ch = curl_init($url);
$options = array(
CURLOPT_CONNECTTIMEOUT =>10,
CURLOPT_USERAGENT => $userAgent,
CURLOPT_AUTOREFERER =>true,
CURLOPT_FOLLOWLOCATION =>true,
CURLOPT_POSTFIELDS => $parameters,// define what you want to post
CURLOPT_RETURNTRANSFER =>true,
CURLOPT_COOKIEFILE => $cookie,
CURLOPT_COOKIEJAR => $cookie ,
CURLOPT_SSL_VERIFYPEER =>0,
CURLOPT_SSL_VERIFYHOST =>0);
curl_setopt_array($ch, $options);
$result = curl_exec($ch);
curl_close($ch);

print($result);

 

The main issue in your code is POST.

The URL you provided in example uses GET to receive parameters, but your script does POST to submit these parameters. As a result, it's natural the output of your script is different than output by original URL.

 

is this code correct with Get method?.... always the same result

$baseurl = "https://www.investing.com/stock-screener/";

$sp = 'country::22|sector::a|industry::a|equityType::a|exchange::9>eq_market_cap;1';
$url = $baseurl.'?sp='.$sp;

$cookie = tmpfile();
$userAgent = 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.31';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$result = curl_exec($ch);
curl_close($ch);
print($result);

 

I just ran the code on my server and yes, it doesn't return the same result as original URL. However, your code is fine, I believe because if I try to open URL like https://www.investing.com/equities/pre-market or similar (without "non-standard" symbols like :: > | in URL), it works perfectly. My guess is that maybe PHP interprets these symbols in URL somehow differently?

 

If you are looking to fetch the result rather than the static page. Then you need to know the result was returned as AJAX, not from a static page.
You can use your browser's Web Developer Tool to see that.

The page is making AJAX POST to

https://www.investing.com/stock-screener/Service/SearchStocks

Code (markup):

The result returned is in JSON like below.

{
   "totalCount": 770,
   "hits": [{
     "pair_ID": 989586,
     "stock_symbol": "MLION",
     "industry_id": 5,
     "avg_volume": 0,
     "avg_volume_eu": "0",
     "pair_change_percent": -70,
     "pair_change_percent_eu": "-70,00",
     "a52_week_high": 0.05,
     "a52_week_high_eu": "0,050",
     "a52_week_low": 0.012,
     "a52_week_low_eu": "0,012",
     "turnover_volume": 86000,
     "turnover_volume_eu": "86.000",
     "last": 0.012,
     "last_eu": "0,012",
     "last_frmt": "0.01",
     "avg_volume_frmt": "0.00",
     "pair_change_percent_frmt": "-70.00",
     "a52_week_high_frmt": "0.05",
     "a52_week_low_frmt": "0.01",
     "turnover_volume_frmt": "86.00K",
     "a52_week_high_diff": -76,
     "a52_week_high_diff_eu": "-76,00",
     "a52_week_low_diff": 0,
     "a52_week_low_diff_eu": "0,00",
     "exchange_trans": "Paris",
     "name_trans": "Lionax International",
     "sector_trans": "Technology",
     "industry_trans": "Electronic Instr. & Controls",
     "viewData": {
       "flag": "France",
       "symbol": "MLION",
       "link": "\/equities\/lionax-international-investment",
       "name": "Lionax International Investment Holdings Ltd"
     },
     "tech_sum_900_constant": "Neutral",
     "tech_sum_1800_constant": "Strong Sell",
     "tech_sum_3600_constant": "Strong Sell",
     "tech_sum_18000_constant": "Strong Sell",
     "tech_sum_86400_constant": "Strong Sell",
     "tech_sum_week_constant": "Strong Sell",
     "tech_sum_month_constant": "Strong Sell",
     "daily": -70,
     "daily_eu": "-70,00",
     "daily_frmt": "-70.00"
   },
   ...
}

Code (markup):

p/s: remember to check the site's TOS for scrapping their result

 

Hi Voon
thank you for your help

is it possible to have the example for the result you show ?

 

Sure. Here's the code with some note in the comment.

$url = "https://www.investing.com/stock-screener/Service/SearchStocks";

//note: parameters to post, followed what's being post in web developer tool
//or if you want to further understand, you need to find out how to turn
//sp=country::22|sector::a|industry::a|equityType::a|exchange::9%3Eeq_market_cap;
//into parameters below
$parameters =
[
  "country[]" => "22",
  "sector" => "7,5,12,3,8,9,1,6,2,4,10,11",
  "industry" => "81,56,59,41,68,67,88,51,72,47,12,8,50,2,71,9,69,45,46,13,94,102,95,58,100,101,87,31,6,38,79,30,77,28,5,60,18,26,44,35,53,48,49,55,78,7,86,10,1,34,3,11,62,16,24,20,54,33,83,29,76,37,90,85,82,22,14,17,19,43,89,96,57,84,93,27,74,97,4,73,36,42,98,65,70,40,99,39,92,75,66,63,21,25,64,61,32,91,52,23,15,80",
  "equityType" => "ORD,DRC,Preferred,Unit,ClosedEnd,REIT,ELKS,OpenEnd,Right,ParticipationShare,CapitalSecurity,PerpetualCapitalSecurity,GuaranteeCertificate,IGC,Warrant,SeniorNote,Debenture,ETF,ADR,ETC,ETN",
  "exchange[]" => "9",
  "pn" => "1",
  "order[col]" => "eq_market_cap",
  "order[dir]" => "a"
];

//note: followed headers sent from web developer tool
$headers =
[
  "Accept: application/json, text/javascript, */*; q=0.01",
  "Accept-Language: en-US,en;q=0.5",
  "X-Requested-With: XMLHttpRequest",
  "Connection: keep-alive",
  "Pragma: no-cache",
  "Cache-Control: no-cache",
];


$cookie = tmpfile();

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.31");
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

//note: almost the same as your original php curl code
//just added below 4 lines to set the request method as POST,
//added a referrer and some post headers
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($parameters));
curl_setopt($ch, CURLOPT_REFERER, "https://www.investing.com/stock-screener/?sp=country::22|sector::a|industry::a|equityType::a|exchange::9%3Eeq_market_cap;1");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$response = curl_exec($ch);
curl_close($ch);

//note: you can just echo $response or decode the json result like below
print_r(json_decode($response, true));

PHP:

 

thank you very much Voon
now i will learn 'ajax request'
Have a good day
Nuno