Hi someone helps I need to write php script for fetching all urls from a particular site for instance site is youtube here i gotta take all video urls what should i do? I don't have bit of idea. Genius and experienced persons give me some idea.
absolutely it's not easy....but if you do not spider all the pages how can you get all the urls?invade google servers? or download from youtube database ...?
Absolutely!! Using open source spiders or crawlers such as Heritrix is a very good option to crawl huge websites such as Youtube, With one hour of video uploaded to Youtube every second its going to be one never ending process. To crawl simple websites with less than 500 pages, free version of "Screaming Frog" can be very helpful.
Thanks very much. But I've been looking for some ideas to write php script. Not other's. If you got any logic regarding this, then tell me. Again thanks to all...
if you want to wirte your own script to spider youtube , i think "PHPQUERY"(https://code.google.com/p/phpquery/) can help you