# Removing duplicates from a list
**URL:** https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874
**Category:** Factual Questions
**Created:** [July 9, 2008, 6:52pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874 "2008-07-09T18:52:28Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@butler1850](https://boards.straightdope.com/u/butler1850)
#### Post date: [July 9, 2008, 6:52pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/1 "2008-07-09T18:52:28Z")
I’m attempting to generate a list of users from a windows event log (system log on a print server), and though I can export the logs, and pull out a list of the users, due to the high amount of traffic on the print server, there are a large number of entries for each user.
The end result that I’m looking for is to get a list, which represents each user only once, so that I can send a bulk email to the users in my environment who use this particular server.
Currently, the exports go out into a .CSV, but I can export in other formats as well.
I do, however, run into the 65K lines limit in Excel (very busy servers).
Anyone have any thoughts?
---
### Author:  [@control-z](https://boards.straightdope.com/u/control-z)
#### Post date: [July 9, 2008, 6:59pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/2 "2008-07-09T18:59:42Z")
You need a sort program. One I’ve used for years is OptTech sort, it easily sorts a 400 megabyte 1.8 million record database. I’m sure there are free sort programs available too.
---
### Author:  [@Leaffan](https://boards.straightdope.com/u/Leaffan)
#### Post date: [July 9, 2008, 8:00pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/3 "2008-07-09T20:00:40Z")
You could open the .csv file in Excel and remove duplicate rows like [this.](http://office.microsoft.com/en-us/excel/HA010346261033.aspx?pid=CL100570551033)
---
### Author:  [@UncleRojelio](https://boards.straightdope.com/u/UncleRojelio)
#### Post date: [July 9, 2008, 8:08pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/4 "2008-07-09T20:08:55Z")
[QUOTE=butler1850]
Anyone have any thoughts?
[/QUOTE]
PERL, PHP, or even a simple shell script using “awk | sort | uniq \< input.csv \> output.txt” will work.
---
### Author:  [@UncleRojelio](https://boards.straightdope.com/u/UncleRojelio)
#### Post date: [July 9, 2008, 8:44pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/5 "2008-07-09T20:44:04Z")
Here it is is PHP:
```auto
//open your files
$inputFile = "InputFile.csv";
$outputFile = "OutputFile.txt";
// create file handles
$ifh = fopen($inputFile, 'r');
$ofh = fopen($outputFile, 'w');
// loop through the input file, stuffing the addresses into and array
while($theData = fgets($ifh)) {
$theChunks = explode(",", $theData);
$addressArray[] = $theChunks[0];
}
// sort the array
ksort($addressArray);
// output the array
foreach($addressArray as $address)
fputs($ofh, $address);
// close the files
fclose($ifh);
fclose($ofh);
```
---
### Author:  [@K364](https://boards.straightdope.com/u/K364)
#### Post date: [July 9, 2008, 11:40pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/6 "2008-07-09T23:40:23Z")
[QUOTE=Leaffan]
You could open the .csv file in Excel and remove duplicate rows like [this.](http://office.microsoft.com/en-us/excel/HA010346261033.aspx?pid=CL100570551033)
[/QUOTE]
I would use a variant on this method - Data/Filter/Advanced filter/Unique Records Only/Copy to another location. Seems simply than selecting/show all/deleting/pasting.
---
### Author:  [@Mangetout](https://boards.straightdope.com/u/Mangetout)
#### Post date: [July 10, 2008, 12:18am UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/7 "2008-07-10T00:18:51Z")
I’d import the list into Access and do a quick query - something like this:
SELECT Table1.username
FROM Table1
GROUP BY Table1.username;
- then export the results to whatever format was appropriate.
---
### Author:  [@J\_Cubed](https://boards.straightdope.com/u/J_Cubed)
#### Post date: [July 10, 2008, 1:57am UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/8 "2008-07-10T01:57:40Z")
Open the original list and a new file. Go through the list and for each username, think back and remember if you’ve seen it before. If not, type it into the new file. If you have already typed it, don’t type it. If you’re really pressed for time, use your computer’s Copy and Paste functions.
Or what **UncleRojelio** said.
---
### Author:  [@MrSquishy](https://boards.straightdope.com/u/MrSquishy)
#### Post date: [July 10, 2008, 2:35am UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/9 "2008-07-10T02:35:18Z")
[QUOTE=Mangetout]
I’d import the list into Access and do a quick query - something like this:
SELECT Table1.username
FROM Table1
GROUP BY Table1.username;
- then export the results to whatever format was appropriate.
[/QUOTE]
I think I can save you a few keystrokes:
```auto
select distinct username from table1;
```
I think that would work in Access. I can’t remember if you use “distinct” or “unique” or either one.
---
### Author:  [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)
#### Post date: [July 10, 2008, 11:57am UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/10 "2008-07-10T11:57:17Z")
If you only need to do this once and the user count is small, even though the record count is large, you could even do the “group by” manually.
Just export the event log as csv, use the Windows comamnd-line Sort app to sort by the user name (enter c:>sort /? for help), then open it in Notepad & scroll to find your distinct users. Notepad will handle millions of lines, and you just lean on the page down key while watching the name column for changes as the records zoom by.
Probably not practical if you have more than a couple dozen users.
---
### Author:  [@Ximenean](https://boards.straightdope.com/u/Ximenean)
#### Post date: [July 10, 2008, 1:02pm UTC](https://boards.straightdope.com/t/removing-duplicates-from-a-list/455874/11 "2008-07-10T13:02:58Z")
Using just Windows command line utilities, and assuming you have an input file containing just the users’ names, like
Roy
Jen
Moss
Jen
Jen
Moss
Rich
Roy
Den
Roy
etc.
You could copy the file to a new temporary directory, change to that directory and do something like
```auto
echo bla>blabla.txt
for /f %a in (input.txt) do copy /y blabla.txt %a
dir /b *. > uniquenames.txt
```
uniquenames.txt would then look like
Roy
Jen
Moss
Rich
Den
Gets tricky if the names have characters like \* in them.