any tool that can filter a list of 20 million blog urls???

I want to unique host and unique url the list, buts its so big its seemingly impossible without cutting it into about 200 files! is there any software i can do this in?

or maybe someone could do it for me in exchange for the huge list?

thanks..

 

Use uniq in Linux

 

I'd hope the list is seperated in some way with either a comma between entries or another delimiting character. You should then be able to import them into a spreadsheet or a database.