Use PHP to strip text file before upload?

I have a script that pulls a tab delimited text file from a remote server to my server and then uploads the file to my database. The problem is that the file is too big to upload fully and it is a shared server so I dont have access to php.ini to change the upload size. It does work but I guess it stops uploading once the limit is reached b/c only about 3/4 of the data ends up in my database. The good news is I only need about a 1/4 of the data from the text file.

Is there a way to only upload part of the text file based on certain criteria?
It sould like a long short but I figured I would ask...

 

What method do you use to read the file?

You could use fopen() with fgets() and only read it line by line, and inset the lines separately. This way you don't need that much memory.

 

This is what I am currently using:

  $fcontents = file ('./IDXDOWNLOADRES.txt'); 

  for($i=0; $i<sizeof($fcontents); $i++) { 
      $line = trim($fcontents[$i], '\t'); 
      $arr = explode("\t", $line); 
      #if your data is comma separated
      # instead of tab separated, 
      # change the '\t' above to ',' 
     
      $sql = "insert into homes values ('". 
                  implode("','", $arr) ."')"; 
      mysql_query($sql);
      echo $sql ."<br>\n";
      if(mysql_error()) {
         echo mysql_error() ."<br>\n";
      } 
}

Code (markup):

I am playing with the fread function right now but I have only just started with that...

 

How big is IDXDOWNLOADRES? And do you get a specific error message (which)?

 

the file is 15 mb (a local MLS file). I have it set as a cron job to run at midnight and I dont get any error emails. I just noticed that the entire contents of the file isnt uploading to the database. When I run the script from my browser I see an error that says this multiple times throughout :

"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near..."

 

Try this:

<?php

set_time_limit(0);

$file = './IDXDOWNLOADRES.txt';
$fp = fopen($file, 'rb');

while (!feof($fp))
{
	$line = fgets($fp);
	$arr = explode("\t", $line);
	$arr = array_map('mysql_real_escape_string', $arr);
	
	$query = "INSERT INTO homes VALUES ('". implode("','", $arr) ."')";
	mysql_query($query) OR die(mysql_error());
}

?>

PHP:

It only reads a single line in the memory at the time, and it also avoids PHP's default timeout. (which could be the reason why you only get it partly)

Oh, and it will also fix the MySQL error you get sometimes.

(Untested)

 

It threw this error...

"Column count doesn't match value count at row 1"

 

Hmm, try:

$line = trim(fgets($fp));

PHP:

If that still doesn't work, do a:

echo "<pre>\n";
print_r($arr);
echo "</pre>\n";

PHP:

... in the loop, and see what output you get.


The error you get means that there are more (or less) values than fields in the database.

 

Yeah, that didnt do it. The output showed 47 fields and my database has 47 fields.

By the way I really appreciate the help.