PHP Scraper/Spider Tutorial or Script

Discussion in 'PHP' started by kampbell411, Jan 15, 2008.

  1. #1
    Hi,
    I am looking for maybe a tutorial or a open script on how to make a spider. I want to learn how to pull stuff like content, images, and urls. Thanks in advance.
     
    kampbell411, Jan 15, 2008 IP
  2. logondotinfo

    logondotinfo Peon

    Messages:
    314
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    0
  3. Aequitas

    Aequitas Guest

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    This will help you.

    class spider
      {
        // This class grabs the content from the sites				
      	function setup()
      	{
      		$cookieJar = 'cookies.txt';			
      		curl_setopt($this->curl,CURLOPT_COOKIEJAR, $cookieJar); 
      		curl_setopt($this->curl,CURLOPT_COOKIEFILE, $cookieJar);
      		curl_setopt($this->curl,CURLOPT_AUTOREFERER,true);
      		curl_setopt($this->curl,CURLOPT_TIMEOUT,30);
      		curl_setopt($this->curl,CURLOPT_CONNECTTIMEOUT,25);
      		curl_setopt($this->curl,CURLOPT_FOLLOWLOCATION,true);
      		curl_setopt($this->curl,CURLOPT_RETURNTRANSFER, true);		
      	}
      	
      	function get($url)
      	{ 
      		$this->curl = curl_init($url);
      		$this->setup();			
      		return $this->request();
      	}
      	
      	function request()
      	{
      		return curl_exec($this->curl);
      	}
      }
    PHP:
     
    Aequitas, Jan 15, 2008 IP
  4. BungeeBones

    BungeeBones Peon

    Messages:
    109
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Ive used http://www.sphider.eu/ and found it pretty good. It is open source too so you have the code to work with and see how he did it.
     
    BungeeBones, Jan 15, 2008 IP
  5. kampbell411

    kampbell411 Peon

    Messages:
    82
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Awesome! Thanks everyone.
     
    kampbell411, Jan 16, 2008 IP