Super simple Php Crawler

phpmyadmin will give you the correct syntax. Key length isn't required for TEXT fields.

And the error is from indexing a text field.

 

It will not only give a correct syntax, it will even create tables. So you don't need to write sql statements to create them manually.

 

OK I fixed the text part in the database. The problem was that you can't use the text fields as an index like you can with a varchar.

It still don't work though. I get the same problem with the text field as I get with the varchar.

I echoed the links and the link that is the problem is the first link to use these symbols ''
like this: 'http://example.com/example'

The errorcode is: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'http://www.e24.se/ego/startsidan_s189/http://bors.e24.se/bors24.se/site/overview' at line 1

Anyone know a solution for this problem?

 

As has been said, you need to mysql_real_escape_string() your sql variables.

 

I added this:
mysql_real_escape_string($linkstring) or die(mysql_error());

still getting the same errormessage: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'http://www.e24.se/ego/startsidan_s189/http://bors.e24.se/bors24.se/site/overview' at line 1

I'm very close from throwing the computer out the window now..this thing just won't work..

 

echo the entire sql and show it here.

 

thank you for posting

 

Ok I managed to get it to write to the database by changing the inserting code to this:

$sql = "INSERT INTO testtable (`thetext`) VALUES ('" .mysql_real_escape_string($linkstring) ."')" ;

mysql_query ( $sql ) or die(mysql_error());

// a line of debug
echo $sql ;

Now the problem is that it puts all the links in the same row as one string. I thought it would recognice that they should be sepparated by the commas after converting the array to a string with the implode-function.

I just need the program to know that each link should go in to a new row and then the database part should finally be solved

 

1. Iterate over links.
something like this:

$links = $snoopy->results;
foreach($links as $linkstring){
  $q = "INSERT INTO testtable SET `thetext` =' ".$linkstring." ' ";
  mysql_query($q);
}

PHP:

OR

2. modify $linkstring to insert several values;

$linkstring = implode("','",$snoopy->results);
$sql = "INSERT INTO testtable (`thetext`) VALUES ('" .$linkstring ."')" ;

PHP:

This is unsafe, but I suppose there are no bad symbols for sql in urls.

 

I am teaching a class now where I wrote some crawler, page rank, and search code as part of the homework. I was interested in writing a crawler and search engine in PHP so I wouldn't have to deploy Tomcat to get Nutch running. The code is reasonably short and documented.

http://www.cs.sjsu.edu/faculty/pollett/174.1.09f/?Hw4.shtml

 

Snoopy is madness lol, thing is it doesn't work on sites that I want it to work on, like to get the page source from megavideo video.. it gives 'fake' information for the user so it can't grab the download link xD