web crawler PHP

Discussion in 'PHP' started by ktsirig, Nov 2, 2006.

  1. #1
    Hello,
    I am supposed to construct a page that searches in specific websites to extract information, like those sites from where you can rent a car for example. There is a form in the site where the user selects some fields (for instance departure and drop-off date), then the data are submitted to the other page that searches 2-3 sites and finds which cars are available on those dates.
    I wanted to ask if there are ready scripts to do that, if not, some hints on how to start.
    I am familiar with PHP forms and data extraction from mysql databases, but when you extract data from other sites, I have no clue how I can begin and deal with it...
     
    ktsirig, Nov 2, 2006 IP
  2. nico_swd

    nico_swd Prominent Member

    Messages:
    4,153
    Likes Received:
    344
    Best Answers:
    18
    Trophy Points:
    375
    #2
    Well I'm not going to do your homework for you, but I can give you a few functions that might help.

    php.net/curl

    Or (if no post data is required)

    php.net/file_get_contents

    And
    php.net/preg_match
    php.net/preg_match_all


    I guess these are the most important.

    (Sorry for the not clickable links, I'm not yet allowed to post some)
     
    nico_swd, Nov 2, 2006 IP
  3. streety

    streety Peon

    Messages:
    321
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #3
    streety, Nov 3, 2006 IP
  4. discoverclips

    discoverclips Peon

    Messages:
    491
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I like to use this:

    function gofetch() {
    $fp = @fsockopen("www.website.com", 80, $errno, $errstr, 30);
    if (!$fp) {
       echo "$errstr ($errno)<br />\n";
    } else {
       $out = "GET /anythingyouwant.html HTTP/1.1\r\n";
    
       $out .= "Host: www.website.com\r\n";
       $out .= "Connection: Close\r\n\r\n";
    
       fwrite($fp, $out);
       while (!feof($fp)) {
    $d = fgets($fp);
    
         if (preg_match("/searching for/i", $d)) {
    echo "I found it :" . $d;
    }
    }
    }
    }
    PHP:
     
    discoverclips, Nov 4, 2006 IP