Remove duplicates except 1st occurrence in txt file

I need a php code example to remove duplicates except 1st occurrence from a txt file (see example1) and save the removed lines (see example2) to a new.txt file.


Example 1
blackdog|black dog
blackfrog|black dog
blackflag|black dog
whitecat|white cat
whiterat|white cat
whitehat|white cat

Example 2
blackdog|black dog
whitecat|white cat

Thanks in advance

 

i dont know the php but if you have excel it has a duplicates filter

 

uh
$lines = explode("\n",$text);
$newtext = "";
$newlines = array();
foreach ($lines as $line) {
$terms = explode("|",$line,2);
if(!in_array($terms[0],$newlines)) {
$newlines[] = $terms[0];
$newtext .= $line."\n";
}
}


untested

 

Thanks but this doesnt work, it only copies the data from $text to $newtext

<?php

function make_content_file($filename,$content,$opentype="w"){
$fp_file = fopen($filename, $opentype);
fputs($fp_file, $content);
fclose($fp_file);
}


$text = file_get_contents("done_bad.txt");
$lines = explode("\n",$text);
$newtext = "";
$newlines = array();
foreach ($lines as $line) {
$terms = explode("|",$line,2);
if(!in_array($terms[0],$newlines)) {
$newlines[] = $terms[0];
$newtext .= $line."\n";
make_content_file("done_good.txt",$newtext);
}
}

?>

 

Let me rephrase what Im trying to do again

Remove lines in a csv file IF the duplicates are found in the 2nd column except first occurrence

 

i didn't test the script, so it might not be working, but I know that you are using it wrong.


<?php

function make_content_file($filename,$content,$opentype="w"){
$fp_file = fopen($filename, $opentype);
fputs($fp_file, $content);
fclose($fp_file);
}


$text = file_get_contents("done_bad.txt");
$lines = explode("\n",$text);
$newtext = "";
$newlines = array();
foreach ($lines as $line) {
$terms = explode("|",$line,2);
if(!in_array($terms[1],$newlines)) {
$newlines[] = $terms[1];
$newtext .= $line."\n";
}
}
make_content_file("done_good.txt",$newtext);

?>


try that

 

Thanks for your help Kyosys, It works great

 

How would I make another file.txt that would save the removed (duplicate) lines?

 

<?php

function make_content_file($filename,$content,$opentype="w"){
$fp_file = fopen($filename, $opentype);
fputs($fp_file, $content);
fclose($fp_file);
}


$text = file_get_contents("done_bad.txt");
$lines = explode("\n",$text);
$newtext = $rest = "";
$newlines = array();
foreach ($lines as $line) {
$terms = explode("|",$line,2);
if(!in_array($terms[1],$newlines)) {
$newlines[] = $terms[1];
$newtext .= $line."\n";
} else {
$rest .= $line."\n";
}
}
make_content_file("done_good.txt",$newtext);
make_content_file("done_bad.txt",$rest);

?>

try that


Oh, and you might want to look into the function file_put_contents(), it does pretty much the same as your make_content_file