PHP Memory Usage

Hi,

I've been working on a script to process a few feeds and add items into the DB.
The XML file I'm trying to load is about 400MB, the script could potentially be loading multiple files of this size eventually.
For some reason I'm getting a memory error come up even though I have run a similar script on a similar server with out any issues.

The error message is: Warning: simplexml_load_file(): (null)/home/demo/generated/xml/2_31_1.xml:1969919: parser error : out of memory error in /home/demo/public_html/scripts/classes/feeds.class.php on line 175

Any ideas why this is happening?

 

Better still, can some one point me in the direction of a good tutorial for XMLReader to actually parse and read the content of a file.
I've been looking for ages and trying to get my head round it, but so far have only really been able to use it for determining the structure of a feed rather than getting the content.

 

In your php.ini file, change these two values to the amount you need:
- memory_limit
- php_value memory_limit

Its safer if you can read your XML file piece by piece though.

 

Hi,

Yeah I'm now using a combination of XMLReader and SimpleXML but not 100% there yet.
I'm getting an error when reaching a node that isn't formatted properly.
I've tried putting in a try/catch in which seems to bypass that problem but then I run into another one.

I wonder if any one knows what is going wrong?

$xml = new XMLReader();
$xml->open($strFileName);

while($xml->read() && $objXML->name !== "Villa");
while($xml->name === "Villa"){
	try{
		$node = new SimpleXMLElement($objXML->readOuterXML());
		/*
		Do something with the information and try and
		save to database
		*/
	}catch(Exception $e){
		// Do something to handle error
	}
	
	$xml->next("Villa");
}

$xml->close();
$xml = null;

PHP:

The error message I'm getting now is this:

Warning: XMLReader::next(): An Error Occured while reading in ###### on line 261

Code (markup):

Line 261 is this line: $xml->next("Villa");

Should I instead be doing something like this?

$xml = new XMLReader();
$xml->open($strFileName);

while($xml->read() && $objXML->name !== "Villa");
while($xml->name === "Villa"){
	try{
		$node = new SimpleXMLElement($objXML->readOuterXML());
	}catch(Exception $e){
		// Do something to handle error
	}
	
	if(!empty($node)){
		/*
		Do something with the information and try and
		save to database
		*/
	}
	
	$xml->next("Villa");
}

$xml->close();
$xml = null;

PHP:

 

An alternative to editing your php.ini file, you can include this line of code at the beginning of your php script

ini_set('memory_limit', '128M');

You can modify memory limit according to your requirements.

 

The OP has said the XML file is 400Mb..... simplexml loads the document into memory, so that will not do.

XMLReader (or one of the many other ways) is definatly the way to go on files that are that big.

Without seeing the XML you are parsing - and i dont fancy trawling 400Mb - it is hard to say really. You might want to start by checking the property you are looking at as well; you can use $xml->nodeType to check this, or even $xml->hasValue;

 

ini_set('memory_limit', '512M');

 

change the file size limit in php.ini may be it help you

 

Have a look at the QueryPath library, which I believe will work well for your purposes. It makes working with XHTML, HTML, XML, RSS and Atom feeds very easy.

Good luck!