PHP Script - Keyword Extractor From URL

Hi Guys! Anyone know of a PHP script which would extract keywords from a URL? For example extracting "What Time Is It?" from the URL "http://www.whattimeisit.com".

 

You mean, grab the title from a page?

2 files, both need to be in the same directory.

functions.php: (cURL class copyright to Keith @ DarkZtar)

<?
function GSB($oString, $oStart, $oEnd, $start = 1) {
$strings = explode($oStart, $oString);
$tempstr = $strings[$start];

$strings = explode($oEnd, $tempstr);
return $strings[0];
}

class cURL {
        /*
        * @author Keith Kurson (delusions@gmail.com)
        * @date October 31, 2006
        * @version 1.0

        */
        /*
        * Headers
        */
        var $headers;
        /*
        * User Agent
        */
        var $user_agent;
        /*
        * Compression
        */
        var $compression;
        /*
        * Cookie File
        */
        var $cookie_file;
        /*
        * Proxy Server
        * ip:port
        */
        var $proxy;
        /*
        * FTP Support
        * Username, Password & Host
        */
        var $ftp_username;
        var $ftp_password;
        var $ftp_host;
        /*
        * Initiate the class
        */
        function cURL($cookies=TRUE,$cookie='cookies.txt',$compression='gzip',$proxy='') {
                $this->error_check();
                $this->headers[] = "Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg";
                $this->headers[] = "Connection: Keep-Alive";
                $this->headers[] = "Content-type: application/x-www-form-urlencoded";
                $this->user_agent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)";
                $this->compression=$compression;
                $this->proxy=$proxy;
                $this->cookies=$cookies;
                if ($this->cookies == TRUE) $this->cookie($cookie);
                /* EDIT FOR FTP HERE */
                $this->ftp_username = '';
                $this->ftp_password = '';
                $this->ftp_host = 'ftp.yoursite.com';
                /* END EDIT */
        }
        /*
        * Error Checks
        */
        function error_check() {
                if (!function_exists('curl_setopt')) $this->Error('You do not hve the <b>cURL</b> package compiled with PHP. Please contact your host.');
                if (!function_exists('ftp_connect')) $this->Error('You do not have the <b>FTP</b> package compiled with PHP. Please contact your host.');
        }
        /*
        * Tests the Cookie File
        */
        function cookie($cookie_file) {
                if (file_exists($cookie_file)) {
                        $this->cookie_file=$cookie_file;
                } else {
                        @$this->ftp_chmod($_SERVER['DOCUMENT_ROOT']) or $this->error('Error with FTP');
                        @fopen($cookie_file,'w') or $this->error("The cookie file could not be opened. Make sure this directory has the correct permissions");
                        $this->cookie_file=$cookie_file;
                        @fclose($cookie_file);
                }
        }
        /*
        * Runs a GET through cURL
        */
        function get($url,$refer='') {
                $process = curl_init($url);
                curl_setopt($process, CURLOPT_REFERER, $refer);
                curl_setopt($process, CURLOPT_HTTPHEADER, $this->headers);
                curl_setopt($process, CURLOPT_USERAGENT, $this->user_agent);
                if ($this->cookies == TRUE) curl_setopt($process, CURLOPT_COOKIEFILE, $this->cookie_file);
                if ($this->cookies == TRUE) curl_setopt($process, CURLOPT_COOKIEJAR, $this->cookie_file);
                curl_setopt($process,CURLOPT_ENCODING , $this->compression);
                curl_setopt($process, CURLOPT_TIMEOUT, 30);
                if ($this->proxy) curl_setopt($cUrl, CURLOPT_PROXY, 'proxy_ip:proxy_port');
                curl_setopt($process, CURLOPT_RETURNTRANSFER, 1);
                $return = curl_exec($process);
                curl_close($process);
                return $return;
        }
        /*
        * Runs a POST through cURL
        */
        function post($url,$data,$refer) {
                $process = curl_init($url);
                curl_setopt($process, CURLOPT_REFERER, $refer);
                curl_setopt($process, CURLOPT_HTTPHEADER, $this->headers);
                curl_setopt($process, CURLOPT_USERAGENT, $this->user_agent);
                if ($this->cookies == TRUE) curl_setopt($process, CURLOPT_COOKIEFILE, $this->cookie_file);
                if ($this->cookies == TRUE) curl_setopt($process, CURLOPT_COOKIEJAR, $this->cookie_file);
                curl_setopt($process, CURLOPT_ENCODING , $this->compression);
                curl_setopt($process, CURLOPT_TIMEOUT, 30);
                if ($this->proxy) curl_setopt($cUrl, CURLOPT_PROXY, 'proxy_ip:proxy_port');
                curl_setopt($process, CURLOPT_POSTFIELDS, $data);
                curl_setopt($process, CURLOPT_RETURNTRANSFER, 1);
                curl_setopt($process, CURLOPT_FOLLOWLOCATION, 1);
                curl_setopt($process, CURLOPT_POST, 1);
                $return = curl_exec($process);
                curl_close($process);
                return $return;
        }
        /*
        * FTP Chmod Script
        * Used if chmod isn't allowed
        */
        function ftp_chmod($path,$mod) {
                $ftp_location = str_replace($path);
                // set up basic connection
                $ftp = ftp_connect($this->ftp_server);

                // login with username and password
                $login_result = ftp_login($ftp, $this->ftp_username, $this->ftp_password);

                // try to chmod $path directory
                if (ftp_site($ftp, 'CHMOD '.$mod.' '.$ftp_location) !== false) {
                $success=TRUE;
                }
                else {
                $success=FALSE;
                }

                // close the connection
                ftp_close($ftp);
                return $success;
        }
        /*
        * Error Output
        */
        function error($error) {
                echo "<center><div style='width:500px;border: 3px solid #FFEEFF; padding: 3px; background-color: #FFDDFF;font-family: verdana; font-size: 10px'><b>cURL Error</b><br>$error</div></center>";
                die;
        }
}

?>

PHP:

page.php: (change the "page.php" in the form tag to whatever you name it)

<?php
include("functions.php");
if(!isset($_POST['weburl'])) {
echo <<<END
<center>
<form action="page.php" method="post">
<b>Web URL to parse:</b> <input type="text" name="weburl">
<br /><input type="submit" value="Parse">
</form></center>
END;
} else {
$cURL = new cURL();
$strHTML = $cURL->get($_POST['weburl']);
$strTitle = GSB($strHTML, "<title>","</title>");
echo $strTitle;
}
?>

PHP:

This only prints the title to the page, but it should work

 

No - that is not what I was looking for...

I'm looking for a function that I can feed URLs and get they keywords that the URL contains.

Examples:
www.whatismyip.com => What Is My Ip
www.facebookads.com => Face Book Ads
www.affiliatewebsite.com => Affiliate Website

 

Anyone know of a solution? Would be greatly appreciated...

 

That would have to be a very smart script with a dictionary array, and even then it might not be 100% accurate. . . My suggestion would be to parse the data from whois.domaintools.com- it displays what it thinks is the "english version" of the URL, right above front page information: http://whois.domaintools.com/bubblecoder.com.

Using the cURL script and clever parsing this can be done.

 

I think Domain tools has systems in place to stop this!

 

It shouldn't be that hard to build one from a dictionary list but the problem is it would not be very accurate like you said. I would build it myself but I figure someone must have already built one like Domain Tools that is freely available.

I thought about going to Domain Tools to parse their data but they have a lot of systems in place to stop people from doing this. Eventually they stop giving you data after like 10 whois per day by ip. I could use proxies but the amount of time spent developing that script might as well be spent towards the dictionary list solution.

 

Just write a 10 lines code and fetch the page ,remove common words

Thats it

Regards

Alex

 

Are you talking about Domain Tools?

Like I said if you request too many times from them in a day they will not return any information. So the system would fail if I needed to Curl them 10,000 times a day unless I used proxies.

And once that system is built all it takes is Domain Tools to block the system from working properly and then I'm back at stage 1.

 

True, I think like you said your energy might be better spent trying to figure out how they went about it. . . They leave it editable so of course their method isn't 100% either. . . It sounds like a fun project actually and their might be a way to do it without a dictionary array. I'm going to attempt this and I'll get back to you on it.

 

I give up. xD

This is like trying to teach a computer english. . .

It was fun to try though

 

Just thought I'd bring this up as well.

Disctionary script would not work.
Let's use experts exchange as an example.
It could be parsed two ways:

1. Experts Exchange
2. Expert Sex Change

Good luck, sorry I couldn't help more.

 

Yeah, one way you might do it would be to parse the keyword density of the url's page and than match that against your url. It wouldn't work 100%, but it would work better than using a dictionary array and it would save memory.

 

@CPURules: Good point!

I am working on a system now that basically will keep reducing the domain by one character until the remaining text matches a dictionary word. For example for ExpertsExchange it would look for:

ExpertsExchange
ExpertsExchang
ExpertsExchan
ExpertsExcha
ExpertsExch
ExpertsExc
ExpertsEx
ExpertsE
Experts - would match a keyword in the dictionary

As soon as it matches a keyword in the dictionary it takes the remaining text and looks in the dictionary for the remainder of the domain to see if it that word is indeed a keyword.

Exchange - would match a keyword in the dictionary

The system would then just stop because there were would be no more characters to match to the dictionary. If there were multiple keywords it would keep looping until no characters match a dictionary keyword.

I haven't fully explained the entire system but that is basically how I vision it. One of my biggest problems is the looping and it essentially takes a longer time then it should.

I will hopefully post something today for you guys to look at. Hopefully one of the geniuses on Digital Point can help me optimize it.

 

Yay

I completed my keyword extractor. It is very crude but I do not want to look at it any more today. I know there is a lot of smart people here on Digital Point so if anyone can think of ways to speed this code up let me know. I will keep trying to optimize the code.

<?php

/**
 * Class: Keyword Extraction
 * Find the keywords from a domain using a dictionary array.
 *
 */

class KeywordExtraction {
  var $current_domain;
  var $no_tld;
  var $count = 0;
  var $keywords;
  var $key_full;
  var $final_keywords;

  function remove_tld() {
  	$this->current_domain = strtolower($this->current_domain);
  	$periods = explode(".", $this->current_domain);
  	for($x=0;$x<count($periods);$x++) {
  		if($periods[$x] == "www" && $x == 0) continue;
  		$result = mysql_query("SELECT COUNT(*) FROM `tlds` WHERE `tld` = '.".$periods[$x]."'");
  		if(mysql_num_rows($result)) {
  		 $row = mysql_fetch_array($result);
  		 if($row[0] == 0 && $x != count($periods)) $this->no_tld .= $periods[$x];
  	  }
  	}
  	$this->all_possible_keys();
  }
  
  function all_possible_keys() {
  	if(empty($this->keywords)) {
  		$this->find_keywords($this->no_tld, $this->count);
  		$this->all_possible_keys();
  		exit;
  	} else {
  		$z=0;
  		$candidate = "";
  		for($x=0;$x<$this->count;$x++) {
  			if(strlen($this->no_tld) != strlen(str_replace(" ", "", $this->keywords[$x]))) {
  				$this->find_keywords(str_replace(str_replace(" ", "", $this->keywords[$x]), "", $this->no_tld), $x);
  				$z++;
  			}
  		}
  	}
  	if($z!=0) {
  	  $this->all_possible_keys();
  	} else {
  	  if(!empty($this->key_full)) {
  	  	$this->final_keywords = $this->key_full[0];
  	  } else {
  	  	$this->final_keywords = $this->keywords[0];
  	  }
  	  echo $this->final_keywords;
  	}
  }
  
  function find_keywords($text, $number) {
  	$text = strtolower($text);
  	$where = '';
  	for($x=0;$x<strlen($text);$x++) {
  		$where .= " `word` = '".substr($text, 0, strlen($text)-$x)."'";
  		if($x != strlen($text)-1) $where .= " OR";
  	}
  	$result = mysql_query("SELECT `word` FROM dictionary WHERE".$where." ORDER BY CHAR_LENGTH(`word`) DESC");
  	if(mysql_num_rows($result) != 0) {
  		$z=0;
	  	while($row = mysql_fetch_row($result)) {
	  	  $row[0] = strtolower($row[0]);
	  	  if(empty($this->keywords[$number])) {
		    $this->keywords[$this->count] = $row[0];
		    $this->count++;
		    $number++;
	  	  } elseif($z == 0 && !empty($this->keywords[$number])) {
	  	  	$lastkeys = $this->keywords[$number];
	  	  	$this->keywords[$number] .= " ".$row[0];
	  	  	if(strlen($this->no_tld) == strlen(str_replace(" ", "", $this->keywords[$number]))) $this->key_full[] = $this->keywords[$number];
	  	  } elseif($z != 0 && !empty($this->keywords[$number])) {
		    $this->keywords[$this->count] = $lastkeys." ".$row[0];
		    if(strlen($this->no_tld) == strlen(str_replace(" ", "", $this->keywords[$this->count]))) $this->key_full[] = $this->keywords[$this->count];
		    $this->count++;
	  	  }
	  	  $z++;
	  	}
	  	
  	} else {
  	  $this->keywords[$number] .= " ".$text;
  	}
  }
}

 $hostname="localhost";
 $username="username";
 $password="password";
 $dbname="databasename";

 mysql_connect($hostname,$username, $password) OR DIE ("Unable to connect to database! Please try again later.");
 mysql_select_db($dbname);
 
 $domain = new KeywordExtraction();
 $domain->current_domain = "IRanAcrossTheMoon.com";
 $domain->remove_tld();
?>

PHP:

For this to work properly you need to populate two tables: Dictionary and TLD

Here is the examples of how mine are setup (I am not going to post all of the dictionary list here its over 300,000 rows):

DROP TABLE IF EXISTS `dictionary`;
CREATE TABLE `dictionary` (
  `word` varchar(24) default NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

LOCK TABLES `dictionary` WRITE;
INSERT INTO `dictionary` VALUES ('A'),('a'),('aa'),('aal'),('aalii'),('aam'),('Aani'),('aardvark');
UNLOCK TABLES

Code (markup):

DROP TABLE IF EXISTS `tlds`;
CREATE TABLE `tlds` (
  `tld` varchar(7) default NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

--
-- Dumping data for table `tlds`
--

LOCK TABLES `tlds` WRITE;
/*!40000 ALTER TABLE `tlds` DISABLE KEYS */;
INSERT INTO `tlds` VALUES ('.arpa'),('.root'),('.aero'),('.asia'),('.biz'),('.cat'),('.com'),('.coop'),('.edu'),('.gov'),('.info'),('.int'),('.jobs'),('.mil'),('.mobi'),('.museum'),('.name'),('.net'),('.org'),('.pro'),('.tel'),('.travel'),('.ac'),('.ad'),('.ae'),('.af'),('.ag'),('.ai'),('.al'),('.am'),('.an'),('.ao'),('.aq'),('.ar'),('.as'),('.at'),('.au'),('.aw'),('.ax'),('.az'),('.ba'),('.bb'),('.bd'),('.be'),('.bf'),('.bg'),('.bh'),('.bi'),('.bj'),('.bm'),('.bn'),('.bo'),('.br'),('.bs'),('.bt'),('.bv'),('.bw'),('.by'),('.bz'),('.ca'),('.cc'),('.cd'),('.cf'),('.cg'),('.ch'),('.ci'),('.ck'),('.cl'),('.cm'),('.cn'),('.co'),('.cr'),('.cu'),('.cv'),('.cx'),('.cy'),('.cz'),('.de'),('.dj'),('.dk'),('.dm'),('.do'),('.dz'),('.ec'),('.ee'),('.eg'),('.er'),('.es'),('.et'),('.eu'),('.fi'),('.fj'),('.fk'),('.fm'),('.fo'),('.fr'),('.ga'),('.gb'),('.gd'),('.ge'),('.gf'),('.gg'),('.gh'),('.gi'),('.gl'),('.gm'),('.gn'),('.gp'),('.gq'),('.gr'),('.gs'),('.gt'),('.gu'),('.gw'),('.gy'),('.hk'),('.hm'),('.hn'),('.hr'),('.ht'),('.hu'),('.id'),('.ie'),('.il'),('.im'),('.in'),('.io'),('.iq'),('.ir'),('.is'),('.it'),('.je'),('.jm'),('.jo'),('.jp'),('.ke'),('.kg'),('.kh'),('.ki'),('.km'),('.kn'),('.kp'),('.kr'),('.kw'),('.ky'),('.kz'),('.la'),('.lb'),('.lc'),('.li'),('.lk'),('.lr'),('.ls'),('.lt'),('.lu'),('.lv'),('.ly'),('.ma'),('.mc'),('.md'),('.me'),('.mg'),('.mh'),('.mk'),('.ml'),('.mm'),('.mn'),('.mo'),('.mp'),('.mq'),('.mr'),('.ms'),('.mt'),('.mu'),('.mv'),('.mw'),('.mx'),('.my'),('.mz'),('.na'),('.nc'),('.ne'),('.nf'),('.ng'),('.ni'),('.nl'),('.no'),('.np'),('.nr'),('.nu'),('.nz'),('.om'),('.pa'),('.pe'),('.pf'),('.pg'),('.ph'),('.pk'),('.pl'),('.pm'),('.pn'),('.pr'),('.ps'),('.pt'),('.pw'),('.py'),('.qa'),('.re'),('.ro'),('.rs'),('.ru'),('.rw'),('.sa'),('.sb'),('.sc'),('.sd'),('.se'),('.sg'),('.sh'),('.si'),('.sj'),('.sk'),('.sl'),('.sm'),('.sn'),('.so'),('.sr'),('.st'),('.su'),('.sv'),('.sy'),('.sz'),('.tc'),('.td'),('.tf'),('.tg'),('.th'),('.tj'),('.tk'),('.tl'),('.tm'),('.tn'),('.to'),('.tp'),('.tr'),('.tt'),('.tv'),('.tw'),('.tz'),('.ua'),('.ug'),('.uk'),('.um'),('.us'),('.uy'),('.uz'),('.va'),('.vc'),('.ve'),('.vg'),('.vi'),('.vn'),('.vu'),('.wf'),('.ws'),('.ye'),('.yt'),('.yu'),('.za'),('.zm'),('.zw');
/*!40000 ALTER TABLE `tlds` ENABLE KEYS */;
UNLOCK TABLES;

Code (markup):

 

Disclaimer on the above code:
It does not work 100% and it never will. I was looking for a solution that works most of the time. The example I used above was for "Iranacrossthemoon.com" and it found the following keywords "Iran across the moon". I am happy with it working about 80%.

I hope other members here can help make the code better!

 

I wrote a tool that does what you're looking for... drewtowers.com/keyword-extraction-utility.html

pm me for the source code

 

Hi,

I tried your website but it didn't work? I typed in Iranacrossthemoon.com but it did not return anything. I am using firefox.

Thanks!

 

I was able to reduce the execution time on this script by 64%. I am done working on this script if anyone else wants to go at it feel free to. In all to do about 100,000 domains would take about 3 minutes and 46 seconds with about an 80% accuracy rate. Not too shabby especially since it is looking through a database of 400,000 dictionary words.

 <?php
 function remove_tld() {
  	$periods = explode(".", $GLOBALS['current_domain']);
  	$strdomain = $GLOBALS['current_domain'];
  	for($x=count($periods)-1;$x>0;$x--) {
  	  $result = mysql_query("SELECT * FROM `tlds` WHERE `tld` = '.".$periods[$x]."'");
  	  if(mysql_num_rows($result) != 0) $strdomain = substr($strdomain, 0, strlen($strdomain)-(strlen($periods[$x])+1));
  	}
  	$GLOBALS['no_tld'] = str_replace("-", "", $strdomain);
  	if(empty($GLOBALS['keywords'])) {
  		find_keywords($GLOBALS['no_tld'], $GLOBALS['count']);
  	}
	
  	for($x=0;$x<$GLOBALS['count'];$x++) {
  		while(strlen($GLOBALS['no_tld']) != strlen($GLOBALS['keywordsnospace'][$x])) {
  			find_keywords(substr($GLOBALS['no_tld'], strlen($GLOBALS['keywordsnospace'][$x])), $x);
  		} 
		if(!empty($GLOBALS['key_full'])) break;
  	}
   	if(!empty($GLOBALS['key_full'])) {
  	  $GLOBALS['final_keywords'] = $GLOBALS['key_full'][0];
  	} else {
  	  $GLOBALS['final_keywords'] = $GLOBALS['keywords'][0];
  	}
  	$GLOBALS['final_keywords'] = str_replace(" s ", "s ", $GLOBALS['final_keywords']);
  	if(substr($GLOBALS['final_keywords'], strlen($GLOBALS['final_keywords'])-2) == " s") $GLOBALS['final_keywords'] = substr($GLOBALS['final_keywords'], 0, strlen($GLOBALS['final_keywords'])-2)."s";
  	return $GLOBALS['final_keywords'];
  }
 
  function find_keywords($text, $number) {
  	$text = strtolower($text);
  	$where = '';
  	for($x=0;$x<strlen($text);$x++) {
  		$where .= " `word` = '".substr($text, 0, strlen($text)-$x)."'";
  		if($x != strlen($text)-1) $where .= " OR";
  	}
  	$result = mysql_query("SELECT `word` FROM dictionary WHERE".$where." ORDER BY CHAR_LENGTH(`word`) DESC");
  	if(mysql_num_rows($result) != 0) {
  		$z=0;
	  	while($row = mysql_fetch_row($result)) {
	  	  $row[0] = strtolower($row[0]);
	  	  if(empty($GLOBALS['keywords'][$number])) {
		    $GLOBALS['keywords'][$GLOBALS['count']] = $row[0];
		    $GLOBALS['keywordsnospace'][$GLOBALS['count']] = $row[0];
		    $GLOBALS['count']++;
		    $number++;
	  	  } elseif($z == 0 && !empty($GLOBALS['keywords'][$number])) {
	  	  	$lastkeys = $GLOBALS['keywords'][$number];
	  	  	$lastkeysnospace = $GLOBALS['keywordsnospace'][$number];
	  	  	$GLOBALS['keywords'][$number] .= " ".$row[0];
	  	  	$GLOBALS['keywordsnospace'][$number] .= $row[0];
	  	  	if(strlen($GLOBALS['no_tld']) == strlen($GLOBALS['keywordsnospace'][$number])) $GLOBALS['key_full'][] = $GLOBALS['keywords'][$number];
	  	  } elseif($z != 0 && !empty($GLOBALS['keywords'][$number])) {
		    $GLOBALS['keywords'][$GLOBALS['count']] = $lastkeys." ".$row[0];
		    $GLOBALS['keywordsnospace'][$GLOBALS['count']] = $lastkeysnospace.$row[0];		    
		    if(strlen($GLOBALS['no_tld']) == strlen($GLOBALS['keywordsnospace'][$GLOBALS['count']])) $GLOBALS['key_full'][] = $GLOBALS['keywords'][$GLOBALS['count']];
		    $GLOBALS['count']++;
	  	  }
	  	  $z++;
	  	}	  	
  	} else {
  	  $GLOBALS['keywords'][$number] .= " ".$text;
  	  $GLOBALS['keywordsnospace'][$number] .= $text;
  	}
  }

 $hostname="";
 $username="";
 $password="";
 $dbname="";

 mysql_connect($hostname,$username, $password) OR DIE ("Unable to connect to database! Please try again later.");
 mysql_select_db($dbname);
$GLOBALS['current_domain'] = "Iranacrossthemooon.com";
print $GLOBALS['current_domain']." => ".remove_tld($GLOBALS['current_domain'])."\n";
?>

PHP:

 

for($x=count($periods)-1;$x>0;$x--) {

Put the count outside of the loop:

$periods_count = count($periods);
for($x = $periods_count - 1; $x > 0; $x--) {

Code (markup):

http://foliovision.com/2007/11/07/phpp-speed-security-code-optimization/