Hi. I am trying to write a code to download each site from textarea form and save it in one file. This code only works for 1 result. What am I missing here? [PHP]<html> <body> <? if(isset($_POST['url'])) { set_time_limit (0); $urls = explode("\n", $_POST['url']); $term = $_POST['term']; foreach ($urls as $url) { $output=''; $output = get_data($url); $myFile = "testFile.txt"; $fh = fopen($myFile, 'a') or die("can't open file"); fwrite($fh, $output); fwrite($fh, "NEXT SITE-------------------------------NEXT SITE"); fclose($fh); } } function get_data($url) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6'); curl_setopt($ch, CURLOPT_HEADER, false); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); $output = curl_exec($ch); curl_close($ch); return $output; } ?> <form action="curl.php" method="post"> URLs: <br/> <textarea rows="20" cols="50" input type="text" name="url" /></textarea><br/> <input type="submit" /> </form> </body> </html> PHP: [/PHP]
Hi, I used test data of these following sites: and got results of following in the text file: NEXT SITE-------------------------------NEXT SITENEXT SITE-------------------------------NEXT SITE<!doctype html> <!--[if lt IE 7 ]> <html class="ie ie6" xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml"> <![endif]--> <!--[if IE 7 ]> <html class="ie ie7" xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml"> <![endif]--> <!--[if IE 8 ]> <html class="ie ie8" xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml"> <![endif]--> <!--[if IE 9 ]> <html class="ie ie9" xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml"> <![endif]--> <!--[if (gt IE 9)|!(IE)]><!--><!-- x --> <html class="" xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml"> <!--<![endif]--> <head> <link href="http://www.runescape.com/css/fonts-50.css" rel="stylesheet" /> <link type="text/css" rel="stylesheet" href="http://www.jagex.com/css/jquery/jquery_jscrollpane_css_2_0_0b11-0.css" /> <link href="http://www.runescape.com/css/global-74.css" rel="stylesheet" /> <link rel="stylesheet" type="text/css" href="http://www.jagex.com/css/shadowbox/shadowbox-0.css"/> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7/jquery.min.js"></script> <script type="text/javascript">window.jQuery||document.write("<script src='http://www.jagex.com/js/jquery/jquery_1_7.js'>\x3C/script>")</script> <script src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.16/jquery-ui.min.js"></script> <script>window.jQuery.ui||document.write("<script src='http://www.jagex.com/js/jquery/jquery_ui_1_8_6.js'>\x3C/script>")</script> <script type="text/javascript" src="http://www.jagex.com/js/modernizr_1_7_min.js"></script> <script type="text/javascript" src="http://www.jagex.com/g=runescape/js/jagex/jagex_cookie-6.js"></script> <script type="text/javascript" src="http://www.jagex.com/js/jquery/jquery_placeholder_1_4.js"></script> <script type="text/javascript" src="http://www.jagex.com/js/jquery/jquery_mousewheel_3_0_6.js"></script> <script>window.JSON||document.write("<script src='http://www.jagex.com/js/json2.js'>\x3C/script>")</script> <script type="text/javascript" src="http://www.jagex.com/js/jquery/jquery_jscrollpane_min_2_0_0b11-0.js"></script> <script type="text/javascript" src="http://www.jagex.com/js/shadowbox/shadowbox-0.js"></script> <script src="http://www.runescape.com/js/global-62.js"></script> <title>RuneScape - MMORPG - The No.1 Free Online Multiplayer Game</title> <meta name="keywords" content="MMORPG, free RPG games online, online multiplayer, role-playing, massively multiplayer games, mmo, RuneScape, Jagex, java" /> <meta name="description" content="Play RuneScape for free, and join a global community of millions as you complete quests and win enormous treasures in a 3D world full of magic and monsters." /> <link rel="icon" type="image/vnd.microsoft.icon" href="http://www.runescape.com/img/global/favicon.ico" /> <link rel="SHORTCUT ICON" href="http://www.runescape.com/img/global/favicon.ico" /> <link rel="apple-touch-icon" href="http://www.runescape.com/img/global/mobile.png"> <meta property="fb:app_id" content="174823375888281" /> <meta property="og:image" content="http://www.runescape.com/img/global/facebook.png" /> <meta property="og:title" content="RuneScape - MMORPG - The No.1 Free Online Multiplayer Game" /> <meta property="og:type" content="game" /> <meta property="og:site_name" content="RuneScape" /> <meta property="og:description" content="Play RuneScape for free, and join a global community of millions as you complete quests and win enormous treasures in a 3D world full of magic and monsters." /> <meta name="google-site-verification" content="RFtGg-G07XweT7nQe032HocvnRPhJAsdZqrq4NX_Op8"> <!--[if lte IE 6]><meta http-equiv="refresh" content="0;url=http://www.runescape.com/unsupported_browser.ws"><![endif]--> <link rel="stylesheet" type="text/css" href="http://www.runescape.com/css/splash_1-58.css"> <!--[if IE 9]><style>.FirstLetter{line-height:44px}</style><![endif]--> <link rel="stylesheet" type="text/css" href="http://www.jagex.com/css/shadowbox/shadowbox-0.css"/> <script src="https://ajax.googleapis.com/ajax/libs/swfobject/2.2/swfobject.js"></script> <script>window.swfobject||document.write("<script src='http://www.jagex.com/js/swfobject/swfobject_2_2.js'>\x3C/script>")</script> <script type="text/javascript" src="http://www.jagex.com/js/shadowbox/shadowbox-0.js"></script> <script type="text/javascript" src="http://www.runescape.com/js/deployJava-1.js"></script> <script id="deployJava" type="text/javascript"> var JAGEX = JAGEX || {}; JAGEX.java = { version: function(){ var versions = deployJava.getJREs(); return versions.length == 0 ? false : versions; }, runescape: function(){ var playable = true; if($.browser.mozilla){ playable = deployJava.versionCheck("1.6.0_31+"); } else{ playable = deployJava.versionCheck("1.6.0_10+"); } return playable; } } </script> <script type="text/javascript" src="http://www.jagex.com/js/jquery/jquery_tools_min.js"></script> <script src='http://www.runescape.com/js/splash_1-54.js'></script> <script type="text/javascript" src="http://www.jagex.com/js/jquery/jquery_thickbox_3.js"></script> <link type="text/css" rel="stylesheet" href="http://www.jagex.com/css/jquery/jquery_thickbox_3-0.css" /> </head> <body> <div id="contentWrapper"> <script type="text/javascript"> if ($("#fb-root").length == 0){ $("body").prepend('<div id="fb-root"></div>'); window.fbAsyncInit = function() { FB.init({ appId: '174823375888281', status: false, cookie: true, xfbml: true, oauth: true }); if($.browser.msie){ FB.UIServer.setLoadedNode = function (a,b) { FB.UIServer._loadedNodes[a.id] = b; } } }; (function() { var e = document.createElement('script'); e.async = true; e.src = document.location.protocol + '//connect.facebook.net/en_US/all.js'; document.getElementById('fb-root').appendChild(e); }()); } </script> <a style="display:none" id="SNbuttonLink" class="thickbox"></a> <script type="text/javascript"> function fbLoginRedirect(buttonName, loginUrl, destUrl, destStr, _target){ $.getJSON('http://services.runescape.com/m=sn-integration/checkLogin.ws?json=?', {"tps":0, "token":FB.getAuthResponse().accessToken, "expiry":FB.getAuthResponse().expiresIn, "signed":FB.getAuthResponse().signedRequest}, function(data) { if(typeof(_target) != 'undefined' && _target != '') $("#SNbuttonLink").attr('target',_target); if (data['sso']<0) { alert("There was a problem verifying your login. Please try again."); } else if (data['sso'].length > 1){ if(data['sna']>-1) { $("#SNbuttonLink").attr('href',loginUrl+'key='+data['sso']+'&tps=0&'+destStr+'&KeepThis=true&TB_iframe=true&height=500&width=560'); $("#SNbuttonLink").click(); $("#TB_iframeContent").attr("allowTransparency", "true"); $("#TB_iframeContent").css("padding-left", "50px"); } else { var loc = destUrl+'?key='+data['sso']+'?'+destStr; if(typeof(_target) != 'undefined' && _target != ''){ window.parent.location = loc; } else { window.location = loc; } } } else { $("#SNbuttonLink").attr('href',loginUrl+'?'+destStr+'&KeepThis=true&TB_iframe=true&height=400&width=442'); $("#SNbuttonLink").click(); $("#TB_iframeContent").attr("allowTransparency", "true"); } }); return false; } </script> <div id="flashWater"></div> <!--[if gte IE 8]><!---><div id="flashFlagsRight"></div><!--<![endif]--> <div id="flashWaterfall"></div> <div id="container"> <h1 class="logo"> <img class="no-counter" src="http://www.runescape.com/img/main/splash/splash1/logo.png?1" id="rs_logo" title="" alt="RuneScape - The World's Greatest Adventure"> </h1> <blockquote class="one"> <p>“RuneScape is the king of the browser.”</p> <p class="cite">Massively.com</p> </blockquote> <blockquote class="two"> <p>“a leading light in the genre”</p> <p class="cite">MMORPG.com</p> </blockquote> <div id="content"> <div id="playWrapper"> <a id="PlayNowWrapperPlayFree" href="https://secure.runescape.com/m=account-creation/create_account_funnel.ws" class="HoverImg GameLink" rel="#popup" onclick="try{pageTracker._trackEvent('Splash', 'Click', 'Play Button 1', null, true);} catch(e){}; TrackingPixel('2002');"> <img src="http://www.runescape.com/img/main/splash/splash1/playnow.png" title="" alt="Play Free Now"> </a> <div id="SN_links" style="display: none"> <a id="PlayNowWrapperPlayFB" href="https://secure.runescape.com/m=sn-integration/facebook/gamelogin.ws?" class="HoverImg" onclick="try{pageTracker._trackEvent('Splash', 'Click', 'Facebook Play Button', null, true);} catch(e){};TrackingPixel('2003');"><img src="http://www.runescape.com/img/main/splash/splash1/facebooklogin.png" alt="Play Now With Facebook" /></a> <script type="text/javascript"> $("#PlayNowWrapperPlayFB").live('click',function(ev){ ev.preventDefault(); if(typeof(FB) == 'undefined' || typeof(FB.getAuthResponse) != 'function'){ /* JAGEX.socialnetworks.SNNotification("Couldn't contact Facebook. Please try again later."); */ } else if(FB.getAuthResponse()==null) { FB.login(function(response) { if(response.authResponse) { // Logged in fbLoginRedirect("PlayNowWrapperPlayFB", "https://secure.runescape.com/m=sn-integration/facebook/gamelogin.ws?","http://www.runescape.com/game.ws","",""); }else{ // No session notification alert("Authentication with Facebook failed. Please try again."); } } , {scope:''}); }else{ // Already has all settings fbLoginRedirect("PlayNowWrapperPlayFB", "https://secure.runescape.com/m=sn-integration/facebook/gamelogin.ws?", "http://www.runescape.com/game.ws","",""); } }); </script> </div> <script type="text/javascript"> $("#SN_links").css("display","block"); </script> </div> <div id="screenshotsVideoWrapper"> <div class="screenshots left"> <a class="HoverImg" target="_blank" rel="shadowbox;gallery=screenshots" href="http://www.runescape.com/img/main/splash/screenshots/screenshot1.jpg"> <img alt="" src="http://www.runescape.com/img/main/splash/splash1/screenie1.jpg"> </a> <a class="HoverImg" target="_blank" rel="shadowbox;gallery=screenshots" href="http://www.runescape.com/img/main/splash/screenshots/screenshot2.jpg"> <img alt="" src="http://www.runescape.com/img/main/splash/splash1/screenie2.jpg"> </a> </div> <div class="video"> <a id="trailerWatch" class="HoverImg" target="_blank" rel="shadowbox;height=720;width=1280" href="http://www.youtube.com/watch?v=ajz5Poh_giI"> <img alt="" src="http://www.runescape.com/img/main/splash/splash1/videoPreview.jpg?2"> </a> </div> <div class="screenshots right"> <a class="HoverImg" target="_blank" rel="shadowbox;gallery=screenshots" href="http://www.runescape.com/img/main/splash/screenshots/screenshot3.jpg"> <img alt="" src="http://www.runescape.com/img/main/splash/splash1/screenie3.jpg"> </a> <a class="HoverImg" target="_blank" rel="shadowbox;gallery=screenshots" href="http://www.runescape.com/img/main/splash/screenshots/screenshot4.jpg"> <img alt="" src="http://www.runescape.com/img/main/splash/splash1/screenie4.jpg"> </a> </a> </div> </div> <div id="content-primary"> <h1 class="FlatHeader">Welcome to RuneScape</h1> <p><span class="FirstLetter">J</span>ourney in the medieval fantasy world of Gielinor and join millions of others in the world's biggest online adventure! Fight, quest, skill or just hang out - the choice is yours in this epic, constantly updated and evolving world.</p> <ul> <li>Totally free to play</li> <li>Browser based - play on any PC any time!</li> <li>Tons of skills to master, hundreds of quests to complete, thousands of monsters to battle</li> <li>Join one of the world's largest online communities</li> </ul> <a id="welcomePlay" rel="#popup" href="https://secure.runescape.com/m=account-creation/create_account_funnel.ws" class="HoverImg GameLink" onclick="try{pageTracker._trackEvent('Splash', 'Click', 'Play Button 2', null, true);} catch(e){}; TrackingPixel('2002');"> <img src="http://www.runescape.com/img/main/splash/welcomeStartPlayingNow.png" title="" alt="Start Playing Now"> </a> <a id="continue" href="http://www.runescape.com/title.ws"onclick=" TrackingPixel('2001');">Continue to full website</a> </div> </div> </div> <div id="Footer" class="Kingthings"> <div class="FooterBottom"> <a href="http://www.jagex.com/g=runescape/p=2sA9c0-ltlLB7H6LqG1ya8HsfouobXJr4Av4RGYIPRo/" id="JagexLogo"><img src="http://www.runescape.com/img/global/logos/jagex.png" alt="Jagex" title="" /></a> <p>This website and its contents are copyright © 1999 - 2012 Jagex Ltd<br />Use of this website is subject to our <a href="http://www.jagex.com/g=runescape/terms/terms.ws" target="_blank">Terms & Conditions</a> and <a href="http://www.jagex.com/g=runescape/privacy/privacy.ws" target="_blank">Privacy Policy</a>. <br><a href="http://www.jagex.com/g=runescape/terms/cookies.ws">Change Cookie Settings</a></p> <div class="LanguageButtonSurround"> <div class="LanguageButton"><img src="http://www.runescape.com/img/global/language_selector/en.png" title="en" alt="en" class="SelectedLang"><img src="http://www.runescape.com/img/global/language_selector/LanguageButtonFg.png" class="foreground"/> <div class="LanguagePanel"> <div class="bottom"> <ul> <li class="flag"><a href="http://www.runescape.com/?set_lang=0" title="English"><img src="http://www.runescape.com/img/global/language_selector/en.png" alt="English" class="flag" /><img src="http://www.runescape.com/img/global/language_selector/LanguageArrow.png" class="LanguageArrow" alt=""></a></li> <li class="flag"><a href="http://www.runescape.com/l=1/?set_lang=1" title="Deutsch"><img src="http://www.runescape.com/img/global/language_selector/de.png" alt="Deutsch" class="flag" /><img src="http://www.runescape.com/img/global/language_selector/LanguageArrow.png" class="LanguageArrow" alt=""></a></li> <li class="flag"><a href="http://www.runescape.com/l=2/?set_lang=2" title="Français"><img src="http://www.runescape.com/img/global/language_selector/fr.png" alt="Français" class="flag" /><img src="http://www.runescape.com/img/global/language_selector/LanguageArrow.png" class="LanguageArrow" alt=""></a></li> <li class="flag"><a href="http://www.runescape.com/l=3/?set_lang=3" title="Português (BR)"><img src="http://www.runescape.com/img/global/language_selector/pt.png" alt="Português (BR)" class="flag" /><img src="http://www.runescape.com/img/global/language_selector/LanguageArrow.png" class="LanguageArrow" alt=""></a></li> </ul> </div> </div> </div> </div> <div id="BottomNavigationFlag"> <div class="fb-like" data-href="http://www.facebook.com/RuneScape" data-send="false" data-layout="button_count" data-width="45" data-show-faces="false" data-colorscheme="dark"></div></div> <div id="SnLinks"> <a href="http://services.runescape.com/m=news/latest_news.rss" title="RSS" target="_blank" class="HoverImg NoFade"><img src="http://www.runescape.com/img/global/logos/MediumRss.png" alt="RSS"/></a> <a href="http://www.facebook.com/RuneScape" title="Follow us on Facebook" target="_blank" class="HoverImg NoFade AddButtons" onclick="try{pageTracker._trackPageview("/outgoing/bookmarking/facebook/http://www.runescape.com/")}catch(x){}; try{_pageTracker._trackPageview("/outgoing/bookmarking/facebook/http://www.runescape.com/")}catch(x){}"><img src="http://www.runescape.com/img/global/social/MediumFacebook.png" alt="Facebook"/></a> <a href="http://twitter.com/RuneScape" title="Follow us on Twitter" target="_blank" class="HoverImg NoFade AddButtons" onclick="try{pageTracker._trackPageview("/outgoing/bookmarking/twitter/http://www.runescape.com/")}catch(x){}; try{_pageTracker._trackPageview("/outgoing/bookmarking/twitter/http://www.runescape.com/")}catch(x){}"><img src="http://www.runescape.com/img/global/social/MediumTwitter.png" alt="Twitter"/></a> <a href="http://www.youtube.com/RuneScape" title="Follow us on YouTube" target="_blank" class="HoverImg NoFade AddButtons" onclick="try{pageTracker._trackPageview("/outgoing/bookmarking/youtube/http://www.runescape.com/")}catch(x){}; try{_pageTracker._trackPageview("/outgoing/bookmarking/youtube/http://www.runescape.com/")}catch(x){}"><img src="http://www.runescape.com/img/global/social/MediumYoutube.png" alt="YouTube"/></a> <div class="CustomPlusOne"> <div class="Mo"></div> <g:plusone annotation="none" href="http://www.runescape.com/"></g:plusone> </div> <script type="text/javascript"> window.___gcfg = {lang: 'en-GB'}; (function(){ var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('head')[0]; s.appendChild(po); })(); </script> </div> </div> </div> <script> var _gaq = _gaq || []; (function(){ var ga = document.createElement('script'); ga.type = "text/javascript"; ga.async = true; ga.src = ("https:" == document.location.protocol?"https://ssl":"http://www")+".google-analytics.com/ga.js"; var s = document.getElementsByTagName('head')[0]; s.appendChild(ga); })(); var javaVer = JAGEX.java.version(); javaVer = (javaVer ? javaVer[0] : "none"); _gaq.push([ "pageTracker._setCustomVar", 1, "Java version", javaVer, 1 ]); _gaq.push([ "_pageTracker._setCustomVar", 1, "Java version", javaVer, 1 ]); _gaq.push( ["pageTracker._setAccount", "UA-2058817-2"], ["pageTracker._setDomainName", "runescape.com"], ["pageTracker._trackPageview", "/splash_first_visit.ws/create=lightbox"] ); var pageTracker = { _trackPageview: function(loc){ _gaq.push(["pageTracker._trackPageview", loc])}, _trackEvent: function(a,b,c,d){ _gaq.push(["pageTracker._trackEvent", a, b, c, d])} }; _gaq.push( ["_pageTracker._setAccount", "UA-2058817-15"], ["_pageTracker._setDomainName", "runescape.com"], ["_pageTracker._trackPageview", "/splash_first_visit.ws/create=lightbox"] ); var _pageTracker = { _trackPageview: function(loc){ _gaq.push(["_pageTracker._trackPageview", loc])}, _trackEvent: function(a,b,c,d){ _gaq.push(["_pageTracker._trackEvent", a, b, c, d])} }; </script> </div> <div id="popup"> <a class="close">Close</a> <iframe frameborder="0" allowtransparency="true" src="https://secure.runescape.com/m=account-creation/create_account_funnel.ws" scrolling="no"></iframe> </div> </body> </html>NEXT SITE-------------------------------NEXT SITE HTML:
This code is for 1 URL at a time so I am guessing when you enter a HTML page and post it. The script will return that page and save it to a .txt file. If you want to do loads of sites at once you will need to 1. Do it manually one at a time remember to give the .txt file a unique name each time or set the permissons to RW+ to add them all to one text file otherwise you will rewrite the old entry each time. 2. Remove the HTML post form and make an array of URLS you wish to send then add a loop for the CURL script to how every many URLs you have to do. You can also use the loop to change the .txt file name each time i.e testFile1.txt testFile2.txt etc
Kevin, I though I did that. As you can see from my code, there is already foreach loop that traverses the array of URLS. Also I have opened testfile.txt for append.
Try this for me please tell me if it shows both sites in your .txt file <? function get_data($url) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6'); curl_setopt($ch, CURLOPT_HEADER, false); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); $output = curl_exec($ch); curl_close($ch); return $output; } $url[0]="http://www.google.com"; $url[1]="http://www.yahoo.com"; // for ($i=0;$i<=1;$i++) { // $output[$i]=get_data($url[$i]); $result=$output[$i]; $myFile = "testFile.txt"; $fh = fopen($myFile, 'a+') or die("can't open file"); fwrite($fh, $result); fwrite($fh, "NEXT SITE-------------------------------NEXT SITE"); fclose($fh); } ?>