Is there any possibility to get all urls of a particular website?

Discussion in 'Google' started by HussainMHB, Apr 29, 2013.

  1. #1
    Hi someone helps
    I need to write php script for fetching all urls from a particular site
    for instance site is youtube
    here i gotta take all video urls
    what should i do?
    I don't have bit of idea.
    Genius and experienced persons give me some idea.
     
    HussainMHB, Apr 29, 2013 IP
  2. sultanofseo

    sultanofseo Notable Member

    Messages:
    9,930
    Likes Received:
    405
    Best Answers:
    0
    Trophy Points:
    265
    #2
    i was thinking writing the php script to fetch all the urls was the idea!
     
    sultanofseo, Apr 30, 2013 IP
  3. pandaplot

    pandaplot Greenhorn

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    why don't you use spiders?like Heritrix WebSPHINX spindle......
     
    pandaplot, May 2, 2013 IP
  4. enous

    enous Well-Known Member

    Messages:
    1,500
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    158
    #4
    The Spiders is a good idea, but if the site is very big like youtube, its a endless task
     
    enous, May 2, 2013 IP
  5. pandaplot

    pandaplot Greenhorn

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #5
    absolutely it's not easy....but if you do not spider all the pages how can you get all the urls?invade google servers? or download from youtube database ...? :confused:
     
    pandaplot, May 2, 2013 IP
  6. pandaplot

    pandaplot Greenhorn

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #6
    哥们你是中国的吧?
     
    pandaplot, May 2, 2013 IP
  7. enous

    enous Well-Known Member

    Messages:
    1,500
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    158
    #7
    :):eek:
     
    enous, May 2, 2013 IP
  8. rankwatch.com

    rankwatch.com Active Member

    Messages:
    898
    Likes Received:
    22
    Best Answers:
    2
    Trophy Points:
    63
    #8
    Absolutely!! Using open source spiders or crawlers such as Heritrix is a very good option to crawl huge websites such as Youtube, With one hour of video uploaded to Youtube every second its going to be one never ending process. To crawl simple websites with less than 500 pages, free version of "Screaming Frog" can be very helpful.
     
    rankwatch.com, May 2, 2013 IP
  9. HussainMHB

    HussainMHB Member

    Messages:
    62
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    38
    #9
    Thanks very much.
    But I've been looking for some ideas to write php script. Not other's. If you got any logic regarding this, then tell me. Again thanks to all...
     
    HussainMHB, May 2, 2013 IP
  10. pandaplot

    pandaplot Greenhorn

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #10
    if you want to wirte your own script to spider youtube , i think "PHPQUERY"(https://code.google.com/p/phpquery/) can help you
     
    pandaplot, May 2, 2013 IP
  11. jimkarter

    jimkarter Notable Member

    Messages:
    5,168
    Likes Received:
    347
    Best Answers:
    0
    Trophy Points:
    235
    #11
    Xenu is the perfect answer. Link is http://home.snafu.de/tilman/xenulink.html
     
    jimkarter, May 3, 2013 IP