Hey I have a question, I'm running a script located here. http://dnfinder.net/rentacoder/alib...ompanies.html&step=3&debug=0&display=0&test=0 It basically crawls the site alibaba for information, but eventually the script will just stop. I looked into the issue and it appeared to be the time limit. I fixed that but now the script is stopping again, but without warning. It seems when the server is making a connection to alibaba and it takes too long to do so, more then 5 seconds, (or for that matter if no data is being sent on the connection to keep it alive within 5 seconds, it just drops.) I have no idea how to deal with this? Can someone point me in the right direction?
Or.. try reviewing the directives in the php.ini file -> http://www.php.net/ini.core You can set the values dynamically using ini_set() -> http://www.php.net/ini_set
I've already seen http://www.php.net/set_time_limit, I should have posted that it has nothing to do with that. That was another problem that I took care of. The problem now is the dropping of the connection if no data is received within a certain time frame. For example make a php program that is doing a bunch of connections to 100's of sites, but DON'T have it display the results as it grabs it. Because nothing is being sent to the users browser while this is happening it closes the connection! Hence my problem. I was looking into the .ini stuff, think this might be it? always_populate_raw_post_data as boolean , I doubt it is, but will play around with it, anyone else have any ideas (try re-creating this yourself).
If you think the problem is that no data is being sent to the user's browser, why not just send something once in a while? Send some spaces and flush the buffer after every few runs through your loop.
I do 'do' that, that fixes the problem for a while, the problem is that I can't send any data while CURL is trying to load a page and it takes to long, since CURL halts progress until it gets what it wants, so how is it possible to keep sending data to that buffer? maybe a CURL_OPTION?
I think the browser terminates connection after {timeout} seconds of inactivity. You can try to insert into script ignore_user_abort(true); PHP: But it seems if browser is disconnected, it will not receive any further output after your Curl task completed. Maybe some headers are exist (KeepAlive xx seconds)..
You can probably do what you want by using CURLOPT_READFUNCTION to channel the data through a custom callback function, but it will be more work. See here: http://curl.haxx.se/libcurl/php/examples/callbacks.html
I think you are onto something, I tried adding the command header('Connection: keep-alive');, but to no avail. The ignore_user_abort sounds interesting, but I need the user to see output , or there is no way of knowing that it finished, unless you have something to propose as a way of figuring that out?
How do I use CURLOPT_READFUNCTION, it only shows me write_function use on the link you sent, not entirely sure, can you show me some code?
Does anyone know how to use CURLOPT_READFUNCTION? Google can't seem to find anything... Thanks in advance!! (Really need help, stuck on job ) I assumed I would do code like this. curl_setopt($curl, CURLOPT_READFUNCTION, 'read_function'); PHP: and for it's function //CURL CALL BACK FUNCTIONS---------------------------------------------------- function read_function($ch,$string) { $length = strlen($string); echo "Received $length bytes<br />\n"; return $length; } PHP: but nothing happened or showed... I then tried write CURLOPT_WRITEFUNCTION but that was just weird, while it outputed data it seems like it screwed up my return transfer, not sure why.
Sorry, I think it is writefunction and not readfunction. It may be the case that the data is sent to your function in lieu of via the return transfer method, so you may need to accumulate it there. It has been a really long time since I've dealt with this so my recollections are a bit hazy (and maybe useless).