1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Site explorer

Discussion in 'PHP' started by StaSen, Sep 11, 2018.

  1. #1
    Hi folks,
    SEMrush
    I am trying to build Site explorer, need to get all website internal and external sites. can anybody help me with a method that get all website's link ( internal and external )
     
    StaSen, Sep 11, 2018 IP
    SEMrush
  2. sarahk

    sarahk iTamer Staff

    Messages:
    25,260
    Likes Received:
    3,438
    Best Answers:
    100
    Trophy Points:
    665
    #2
    Just do it manually and use Xenu
     
    sarahk, Sep 11, 2018 IP
    StaSen likes this.
  3. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #3
    I am trying to build something like Xenu to extract the internal and external links
     
    StaSen, Sep 11, 2018 IP
  4. sarahk

    sarahk iTamer Staff

    Messages:
    25,260
    Likes Received:
    3,438
    Best Answers:
    100
    Trophy Points:
    665
    #4
    It's not hard. What CMS are you targetting?
     
    sarahk, Sep 11, 2018 IP
  5. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #5
    I am not targeting CMS, targeting all kind of websites. need to know the formula that it should get and extract all internal and external links.
     
    StaSen, Sep 11, 2018 IP
  6. sarahk

    sarahk iTamer Staff

    Messages:
    25,260
    Likes Received:
    3,438
    Best Answers:
    100
    Trophy Points:
    665
    #6
    Just use domDocument on each page you get

    http://php.net/manual/en/class.domdocument.php

    If you were targetting a cms then you could hit the database and save bandwidth but if you're going to work on any site then you'll need to use curl to get every page and work through it.
     
    sarahk, Sep 11, 2018 IP
    StaSen likes this.
  7. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #7
    gonna check curl method !

    Thanks for your help, gonna update if I need something else
     
    StaSen, Sep 11, 2018 IP
  8. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #8
    The point that I need to know how I can get all website links programmatically.

    Waiting your response,
    Best Regards,
    Stasen
     
    StaSen, Sep 11, 2018 IP
  9. sarahk

    sarahk iTamer Staff

    Messages:
    25,260
    Likes Received:
    3,438
    Best Answers:
    100
    Trophy Points:
    665
    #9
    I'm really confused
    You've done your homework, right? and looked at tools like http://www.sphider.eu/index.php and decided not to adapt an open source project to your requirements, but you're asking questions as if you've never touched a line of code before

    • use curl to get the page
    • retrieve links from that page
    • work through the links to identify dupes and fresh links
    • go through the fresh links to find those that are internal and use curl to get the page...
     
    sarahk, Sep 11, 2018 IP
  10. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #10
    Thanks for your help, I really appreciate that
     
    StaSen, Sep 11, 2018 IP
  11. JEET

    JEET Well-Known Member

    Messages:
    2,275
    Likes Received:
    118
    Best Answers:
    2
    Trophy Points:
    185
    #11
    Do a search on "phpclasses" There are PHP classes which do this
     
    JEET, Sep 17, 2018 IP
  12. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #12
    Thanks , gonna do that.
     
    StaSen, Sep 17, 2018 IP
  13. StaSen

    StaSen Notable Member

    Messages:
    1,151
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    205
    #13
    Found good method in phpclass, thanks !
     
    StaSen, Sep 18, 2018 IP
  14. jitendra05

    jitendra05 Well-Known Member

    Messages:
    754
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    130
    #14
    This person scammed me and did not pay me. Now he is refusing to even recognize me. I sent him proofs and all discussions. I also sent his product details, emails where i sent reports earlier. He said his paypal got limited, so i waited for one month but now he is not even replying me. So be careful before making any deal with him.
     
    jitendra05, Nov 15, 2018 IP