web crawler

Discussion in 'PHP' started by nandha, Aug 6, 2008.

  1. #1
    hi
    anyone help me (i want send the url on proxy using php sript ) i am using

    $cUrl = curl_init();
    curl_setopt($cUrl, CURLOPT_URL, 'google.com');
    curl_setopt($cUrl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($cUrl, CURLOPT_HTTPPROXYTUNNEL, 1);
    curl_setopt($this->cUrl, CURLOPT_PROXY, 'proxy_name_or_ip:proxy_prot');
    curl_setopt($this->cUrl, CURLOPT_PROXYUSERPWD, 'username:password');
    $PageContent = curl_exec($cUrl);
    curl_close($cUrl);
    this script but this one is not working so any one help php web crawler using proxy script
     
    nandha, Aug 6, 2008 IP
  2. cornetofreak

    cornetofreak Peon

    Messages:
    170
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #2
    TRY USING MY FUNCTION

    function curl_grab_page($site,$proxy,$proxystatus){
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
        if ($proxystatus == 'on') {
            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE); // change your ssl if true
            curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
            curl_setopt($ch, CURLOPT_PROXY, $proxy);
        }
        curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt"); // if you are using a cookie
        curl_setopt($ch, CURLOPT_URL, $site);
        ob_start();      // prevent any output
        return curl_exec ($ch); // execute the curl command
        ob_end_clean();  // stop preventing output
        curl_close ($ch);
    }
    PHP:
    USE LIKE THIS
    curl_grab_page("http://www.google.com","","off");
    PHP:
    this works :)
     
    cornetofreak, Aug 6, 2008 IP
  3. wmtips

    wmtips Well-Known Member

    Messages:
    601
    Likes Received:
    70
    Best Answers:
    1
    Trophy Points:
    150
    #3
    Strange function. Lines after return will never be executed. So you have ob_start without end, and non-closed curl handle.
     
    wmtips, Aug 6, 2008 IP