# Purging duplicates from a long list of e-addresses
**URL:** https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270
**Category:** Factual Questions
**Created:** [October 19, 2009, 1:27am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270 "2009-10-19T01:27:49Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@stuyguy](https://boards.straightdope.com/u/stuyguy)
#### Post date: [October 19, 2009, 1:27am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/1 "2009-10-19T01:27:49Z")
I hold and organize various local activities. Over the years my email list has grown to nearly 2000 e-addresses. The list exists as continuous text in an MS Word document – just one address after another separated by a semi-colon and a space.
I suspect that there are duplicates in the pile. How can I find them and eliminate them?
One idea I had was to convert the text list into a table, then sort (alphabetize) the table. That would make spotting duplicates relatively easy, but it would still involve manual work – reading down the whole list – that I don’t want to do. Is there an easier, automated way?
Thanks all, in advance.
---
### Author:  [@Reply](https://boards.straightdope.com/u/Reply)
#### Post date: [October 19, 2009, 1:40am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/2 "2009-10-19T01:40:31Z")
Do you have Excel?
If so, just copy the whole list into Excel and use the Remove Duplicates function. One click, done. It’s under the Data tab in 2007, probably somewhere else in the other versions.
---
### Author:  [@Reply](https://boards.straightdope.com/u/Reply)
#### Post date: [October 19, 2009, 1:44am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/3 "2009-10-19T01:44:07Z")
I missed the part where you said they’re separated by semicolons. In that case, you need to perform one extra step: Use the excel import wizard and specify the semicolon as a delimiter.
---
### Author:  [@stuyguy](https://boards.straightdope.com/u/stuyguy)
#### Post date: [October 19, 2009, 2:03am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/4 "2009-10-19T02:03:29Z")
Yes, I have Excel. This sounds perfect, but I can seem to locate the “Remove Duplicates” function. Where is that?
---
### Author:  [@Reply](https://boards.straightdope.com/u/Reply)
#### Post date: [October 19, 2009, 3:57am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/5 "2009-10-19T03:57:53Z")
Hmm, maybe that’s a 2007-only function.
If you have 2003, there’s a similar procedure:
[http://www.lytebyte.com/2008/10/30/how-to-remove-duplicates-in-excel-2003/](http://www.lytebyte.com/2008/10/30/how-to-remove-duplicates-in-excel-2003/)
---
### Author:  [@stuyguy](https://boards.straightdope.com/u/stuyguy)
#### Post date: [October 19, 2009, 4:57am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/6 "2009-10-19T04:57:17Z")
Thanks for all your help, Reply!
---
### Author:  [@DanBlather](https://boards.straightdope.com/u/DanBlather)
#### Post date: [October 19, 2009, 7:21am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/7 "2009-10-19T07:21:09Z")
I miss Unix:
$ cat addressfile | sort | uniq \>NoDupAddressFile
---
### Author:  [@psychonaut](https://boards.straightdope.com/u/psychonaut)
#### Post date: [October 19, 2009, 10:37am UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/8 "2009-10-19T10:37:03Z")
> [@DanBlather](#):
>
> I miss Unix:
>
> $ cat addressfile | sort | uniq \>NoDupAddressFile
There’s a few problems with that… first, you’ve got a redundant cat and uniq. Second, e-mail addresses are case-insensitive. A better approach:
$ sort --ignore-case --unique addressfile \> NoDupAddressFile
---
### Author:  [@CalMeacham](https://boards.straightdope.com/u/CalMeacham)
#### Post date: [October 19, 2009, 12:00pm UTC](https://boards.straightdope.com/t/purging-duplicates-from-a-long-list-of-e-addresses/514270/9 "2009-10-19T12:00:15Z")
Next Week on MST3K – _The Human Un-Duplicators!!_