I need a web crawler tool -- please help

Discussion in 'Products & Tools' started by compar, Jan 19, 2006.

0
  1. #1
    I'm looking for a tool or tools that will crawl a site and show you what the SE's spiders see. I use to have several bookmarked, but when my laptop got stolen I lost them all.

    Thanks for you help.
     
    compar, Jan 19, 2006 IP
  2. RyanBlank

    RyanBlank Guest

    Messages:
    93
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    RyanBlank, Jan 19, 2006 IP
  3. DavidAusman

    DavidAusman Peon

    Messages:
    399
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #3
    DavidAusman, Jan 19, 2006 IP
  4. ihatebeans

    ihatebeans Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    hmm i may use one of these thanks guys
     
    ihatebeans, Jan 19, 2006 IP
  5. WebtoolMaster

    WebtoolMaster Active Member

    Messages:
    365
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    53
    #5
    WebtoolMaster, Jun 5, 2010 IP
  6. najwan

    najwan Greenhorn

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #6
    thanks for your sharing...
     
    najwan, Jun 6, 2010 IP
  7. aeroz1

    aeroz1 Active Member

    Messages:
    492
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #7
    I'm using win2k, I have php and mysql working fine w/"localhost" apache. I've tried to install phpdig but haven't gotten anywhere - I can't even begin to understand the install. Having trouble finding other PHP spiders. Are there any others with good support and documentation?
     
    aeroz1, Jun 8, 2010 IP
  8. magicman85

    magicman85 Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    try out httrack
     
    magicman85, Jun 17, 2010 IP
  9. trustedfire9

    trustedfire9 Active Member

    Messages:
    57
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    58
    #9
    hmm , i think u can use Acunetix Web Vulnerability Scanner .
     
    trustedfire9, Jun 19, 2010 IP
  10. Sn!per

    Sn!per Guest

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Check PageRank for all outgoing links from an URL with the powerful PageRank Crawler prdisplay.com/pagerank-crawl.php
     
    Sn!per, Jun 19, 2010 IP
  11. Blue Star Ent.

    Blue Star Ent. Well-Known Member

    Messages:
    1,989
    Likes Received:
    31
    Best Answers:
    0
    Trophy Points:
    160
    #11
    Someone started a thread about making their own spider script. It is located here : LINK
     
    Blue Star Ent., Jun 19, 2010 IP