Hi Im working on a seo project and I have created an html page with over 3000 links to web directories. Many of these directories are listed more than once. I need a script/software/anything to simply: Scan the page - Find Duplicated Links - Remove Them or at least Mark them. Any help would be appreciated. Thank You.
GSitecrawler might do the trick - its a sitemap builder but uses a bot and give error and dupe reports.