1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Error : send parameters with php curl

Discussion in 'PHP' started by nunonline, Dec 25, 2017.

  1. #1
    Hello

    i'am trying to send parameters via php curl in a stock screener to have the result in this page:
    https://www.investing.com/stock-screener/?sp=country::22|sector::a|industry::a|equityType::a|exchange::9%3Eeq_market_cap;1

    with the code i have the stock screner page but it's not the same at the page above: can someone help me? thank

    $url ="https://www.investing.com/stock-screener/";
    $parameters ="sp=country::22|sector::a|industry::a|equityType::a|exchange::9>eq_market_cap;1";

    $cookie = tmpfile();
    $userAgent ='Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.31';
    $ch = curl_init($url);
    $options = array(
    CURLOPT_CONNECTTIMEOUT =>10,
    CURLOPT_USERAGENT => $userAgent,
    CURLOPT_AUTOREFERER =>true,
    CURLOPT_FOLLOWLOCATION =>true,
    CURLOPT_POSTFIELDS => $parameters,// define what you want to post
    CURLOPT_RETURNTRANSFER =>true,
    CURLOPT_COOKIEFILE => $cookie,
    CURLOPT_COOKIEJAR => $cookie ,
    CURLOPT_SSL_VERIFYPEER =>0,
    CURLOPT_SSL_VERIFYHOST =>0);
    curl_setopt_array($ch, $options);
    $result = curl_exec($ch);
    curl_close($ch);
    SEMrush
    print($result);
     
    Last edited: Dec 25, 2017
    nunonline, Dec 25, 2017 IP
    SEMrush
  2. phpmillion

    phpmillion Member

    Messages:
    145
    Likes Received:
    11
    Best Answers:
    4
    Trophy Points:
    45
    #2
    The main issue in your code is POST.

    The URL you provided in example uses GET to receive parameters, but your script does POST to submit these parameters. As a result, it's natural the output of your script is different than output by original URL.
     
    phpmillion, Dec 26, 2017 IP
  3. nunonline

    nunonline Greenhorn

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    is this code correct with Get method?.... always the same result :(

    $baseurl = "https://www.investing.com/stock-screener/";

    $sp = 'country::22|sector::a|industry::a|equityType::a|exchange::9>eq_market_cap;1';
    $url = $baseurl.'?sp='.$sp;

    $cookie = tmpfile();
    $userAgent = 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.31';
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
    curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
    curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
    $result = curl_exec($ch);
    curl_close($ch);
    print($result);
     
    nunonline, Dec 26, 2017 IP
  4. phpmillion

    phpmillion Member

    Messages:
    145
    Likes Received:
    11
    Best Answers:
    4
    Trophy Points:
    45
    #4
    I just ran the code on my server and yes, it doesn't return the same result as original URL. However, your code is fine, I believe because if I try to open URL like https://www.investing.com/equities/pre-market or similar (without "non-standard" symbols like :: > | in URL), it works perfectly. My guess is that maybe PHP interprets these symbols in URL somehow differently?
     
    phpmillion, Dec 27, 2017 IP
  5. voon

    voon Active Member

    Messages:
    23
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    63
    #5
    If you are looking to fetch the result rather than the static page. Then you need to know the result was returned as AJAX, not from a static page.
    You can use your browser's Web Developer Tool to see that.

    The page is making AJAX POST to
    https://www.investing.com/stock-screener/Service/SearchStocks
    Code (markup):
    The result returned is in JSON like below.
    
    {
       "totalCount": 770,
       "hits": [{
         "pair_ID": 989586,
         "stock_symbol": "MLION",
         "industry_id": 5,
         "avg_volume": 0,
         "avg_volume_eu": "0",
         "pair_change_percent": -70,
         "pair_change_percent_eu": "-70,00",
         "a52_week_high": 0.05,
         "a52_week_high_eu": "0,050",
         "a52_week_low": 0.012,
         "a52_week_low_eu": "0,012",
         "turnover_volume": 86000,
         "turnover_volume_eu": "86.000",
         "last": 0.012,
         "last_eu": "0,012",
         "last_frmt": "0.01",
         "avg_volume_frmt": "0.00",
         "pair_change_percent_frmt": "-70.00",
         "a52_week_high_frmt": "0.05",
         "a52_week_low_frmt": "0.01",
         "turnover_volume_frmt": "86.00K",
         "a52_week_high_diff": -76,
         "a52_week_high_diff_eu": "-76,00",
         "a52_week_low_diff": 0,
         "a52_week_low_diff_eu": "0,00",
         "exchange_trans": "Paris",
         "name_trans": "Lionax International",
         "sector_trans": "Technology",
         "industry_trans": "Electronic Instr. & Controls",
         "viewData": {
           "flag": "France",
           "symbol": "MLION",
           "link": "\/equities\/lionax-international-investment",
           "name": "Lionax International Investment Holdings Ltd"
         },
         "tech_sum_900_constant": "Neutral",
         "tech_sum_1800_constant": "Strong Sell",
         "tech_sum_3600_constant": "Strong Sell",
         "tech_sum_18000_constant": "Strong Sell",
         "tech_sum_86400_constant": "Strong Sell",
         "tech_sum_week_constant": "Strong Sell",
         "tech_sum_month_constant": "Strong Sell",
         "daily": -70,
         "daily_eu": "-70,00",
         "daily_frmt": "-70.00"
       },
       ...
    }
    
    Code (markup):
    p/s: remember to check the site's TOS for scrapping their result :)
     
    voon, Jan 17, 2018 IP
  6. nunonline

    nunonline Greenhorn

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #6
    Hi Voon
    thank you for your help

    is it possible to have the example for the result you show ?
     
    Last edited: Jan 21, 2018
    nunonline, Jan 21, 2018 IP
  7. voon

    voon Active Member

    Messages:
    23
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    63
    #7
    Sure. Here's the code with some note in the comment.

    
    $url = "https://www.investing.com/stock-screener/Service/SearchStocks";
    
    //note: parameters to post, followed what's being post in web developer tool
    //or if you want to further understand, you need to find out how to turn
    //sp=country::22|sector::a|industry::a|equityType::a|exchange::9%3Eeq_market_cap;
    //into parameters below
    $parameters =
    [
      "country[]" => "22",
      "sector" => "7,5,12,3,8,9,1,6,2,4,10,11",
      "industry" => "81,56,59,41,68,67,88,51,72,47,12,8,50,2,71,9,69,45,46,13,94,102,95,58,100,101,87,31,6,38,79,30,77,28,5,60,18,26,44,35,53,48,49,55,78,7,86,10,1,34,3,11,62,16,24,20,54,33,83,29,76,37,90,85,82,22,14,17,19,43,89,96,57,84,93,27,74,97,4,73,36,42,98,65,70,40,99,39,92,75,66,63,21,25,64,61,32,91,52,23,15,80",
      "equityType" => "ORD,DRC,Preferred,Unit,ClosedEnd,REIT,ELKS,OpenEnd,Right,ParticipationShare,CapitalSecurity,PerpetualCapitalSecurity,GuaranteeCertificate,IGC,Warrant,SeniorNote,Debenture,ETF,ADR,ETC,ETN",
      "exchange[]" => "9",
      "pn" => "1",
      "order[col]" => "eq_market_cap",
      "order[dir]" => "a"
    ];
    
    //note: followed headers sent from web developer tool
    $headers =
    [
      "Accept: application/json, text/javascript, */*; q=0.01",
      "Accept-Language: en-US,en;q=0.5",
      "X-Requested-With: XMLHttpRequest",
      "Connection: keep-alive",
      "Pragma: no-cache",
      "Cache-Control: no-cache",
    ];
    
    
    $cookie = tmpfile();
    
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_HEADER, false);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.31");
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    
    //note: almost the same as your original php curl code
    //just added below 4 lines to set the request method as POST,
    //added a referrer and some post headers
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($parameters));
    curl_setopt($ch, CURLOPT_REFERER, "https://www.investing.com/stock-screener/?sp=country::22|sector::a|industry::a|equityType::a|exchange::9%3Eeq_market_cap;1");
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    
    $response = curl_exec($ch);
    curl_close($ch);
    
    //note: you can just echo $response or decode the json result like below
    print_r(json_decode($response, true));
    
    PHP:
     
    voon, Jan 21, 2018 IP
  8. nunonline

    nunonline Greenhorn

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #8
    thank you very much Voon
    now i will learn 'ajax request'
    Have a good day
    Nuno
     
    Last edited: Jan 22, 2018
    nunonline, Jan 22, 2018 IP