I am trying to create domains list for selected .tld and then check them for some metrics. Is there any software which I could use? Ideal software abilities: - multithreaded (can use as much as possible from CPU and bandwidth) - I will set one website (or few websites) as start position for that software - for example something like DMOZ.org - software will start to crawl them and will try to find other domain names used in a href tags and those domains will add to batch for checking - and this will continue until there wont be any links pointed to new domain names - software should load only html without any assets (to save bandwidth) - option for filtering domains by tld (so I can process only domains with selected tld) - export capabilities Forgive me, if there is similiar thread here, but I cant figure out what I should try to find.
Hi, if you are serious about this project (i.e. not "looking for $10 script"), feel free to contact me, we specialize on creating robots.