Dear DP members, For something I am creating i need a huge URL list, it does not matter what kinds of urls are in the list as long as they are all unique. by this i mean not bladiebla.com bladiebla.com/bla bladiebla.com/bla2 etc. but bla.com bladie.com bladiebla.com any list would help a great deal in my project. thanks in advance. john.
why not built a quick app or script - that builds it in a brute force manner? As in add "a.com" to the list, have script verify it exists, then "b.com", verify it exists, and so on - this should build a monstrous list quick without leaving any out (depending on how long you let it run)
Glad it helped... you could probably do it pretty quick too if you just verified the domain was registered instead of screen scrape the page to verify it exists and has content.
I've read on several places all 4 letter domain names are pretty much took up, there's not many left out there. So if you do a brute force of numbers and letters on at least 4 letter level, that's about 10,000+ domains easy.