Need a dupe remover

Discussion in 'Products & Tools' started by michael11, Jan 29, 2007.

0
  1. #1
    Hi,

    now I need a dupe remover which does the following:

    Scan a .txt file and delete duplicate lines of exact same content. Reason is, I created huge keyword lists for Adwords campaigns, it goes in the many hundreds for each list. I need to merge the different lists to one list but this means I'd have tons of duplicate keyphrases listed.

    In the txt lists it's always 1 keyphrase per line. The tool should only delete duplicate lines and not duplicate keywords.

    It's actually a simple tool but I cannot find it anywhere on Google etc. Tons of dupe removers which delete dupe files, emails etc on your HD are listed but nothing I'm looking for.

    If someone has an idea would be very much appreciated.

    Thanks
    Michael
     
    michael11, Jan 29, 2007 IP
  2. lmoss

    lmoss Peon

    Messages:
    32
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    lmoss, Jan 29, 2007 IP
  3. michael11

    michael11 Peon

    Messages:
    234
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Nope, XP only here :)
     
    michael11, Jan 29, 2007 IP
  4. lmoss

    lmoss Peon

    Messages:
    32
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You can install Cygwin to run Unix commands on Windows: http://www.cygwin.com/

    Or maybe you can access a Unix shell at your web host?
     
    lmoss, Jan 29, 2007 IP
  5. michael11

    michael11 Peon

    Messages:
    234
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Any more ideas?
     
    michael11, Jan 31, 2007 IP
  6. eljefe3

    eljefe3 Peon

    Messages:
    894
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Doesn't adwords do this for you? I don't know but I would think they would be able to filter this out if you are using one word per line. If not there are tools out there. Think I bought one for like $29 or so, but can't remember the name of the program.
     
    eljefe3, Jan 31, 2007 IP
  7. Pat Gael

    Pat Gael Banned

    Messages:
    1,331
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I have not the name of the software at hand, but there is a bulk email sofware that include tools for removing duplicate emails (and text lines) from a text.

    Otherwise I would suggest return to the basics, do a researd for MS-DOS remove dupe and you will find useful utilities that work under a Windows box.
     
    Pat Gael, Jan 31, 2007 IP
  8. Paintball Guns

    Paintball Guns Active Member

    Messages:
    173
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    60
    #8
    Notepad++ does this. In Notepad++, go to this menu option:
    TextFX > TextFX Tools > +Sort outputs only UNIQUE
    Then go to this menu option: TextFX > TextFX Tools > +Sort lines case insensitive
    The result is a unique list with all duplicate lines removed.

    Here is a very simple online tool (probably too simple): http://jura.wi.mit.edu/bioc/tools/compare.php

    Here is a more robust online tool: http://www.esqsoft.com/tools/dedupe-list.htm

    Another online tool, made especially for cleaning lists of keywords: http://www.servermind.com/keyword-list-cleaner.html
     
    Last edited: Aug 4, 2010
    Paintball Guns, Aug 4, 2010 IP