Ask MetaFilter questions tagged with duplicate

Cleaning up messy music folders

1024x768 — Sun, 11 Oct 2009 10:11:46 -0800

Help me clean up my messy music files I am embarking on a clean up of my enormously messy MP3 folders. After several computer switches, and crash+restore jobs I have approximately a dozen different folders containing between hundreds and thousands of MP3s each. The issue is each folder is at least a partial duplicate of several others. Each of these dozen folders contains subfolders in various organizational formats. There's far too many files for me to devote the time to manually eliminating duplicates. Confounding matters slightly is that many of the MP3s have woefully inadequate tagging/ID3 data.

This motley collection lives on a Macbook pro running 10.5. I'd like to find an app/script/method that does the following

1) Eliminates all the dupes
2) Cleans up the ID3 info

The cleaned up collection will then be put into an iTunes library which I will let keep things from getting too messy in the future. I would prefer to use iTunes due to ease of syncing with my iPhone, but if there is a truly compelling alternative that will take care of everything, I am willing to consider it.

I'd prefer a free method of doing this, but would shell out a little bit of scratch if there were an app out there that did exactly what I wanted.

osx applemail imap mbox emlx duplicate emails email messages

DLWM — Thu, 16 Apr 2009 06:33:51 -0800

If I make changes using a command-line utility to an mbox file that is the local cache of an IMAP account, how can I get those changes uploaded to the server? (Mac OS X 10.4, Mail.app, Perl script.) I use Mail.app for my Gmail and work email, both of which use IMAP. I have a local cache of everything that gets synced when I'm connected to the internet.

I have a bunch of duplicate messages within both of these accounts. I found a Perl script that will go through all the emlx files in each local mbox directory (i.e., the local cache) and delete dupes. But I'm thinking that when I connect again and Mail.app sees that there's a bunch of messages on the server that aren't in my local cache, it'll just download them again.

How can I get Mail.app to tell the servers that the messages that are missing on the local copy should be deleted from the server copy, too?

I've tried using this AppleScript that deletes the messages within Mail.app, but it's way to slow on my machine. The command-line Perl script is so much faster.

Copy-Down in SQL

rokusan — Tue, 14 Apr 2009 22:36:40 -0800

Is there a GUI SQL editor that emulates a spreadsheet-style "copy down"? If not, is there a technique I should learn? I seem to spend 10-15 mins a day touching up some mysql data in a huge remote CRM-type database (I don't have shell access) and I notice that I spend most of that time copy-pasting the same corrections down one column for 50 or 60 rows. (I do a select to find new stuff, eyeball it, and find that one of the records has the right info and the other fifty need to be made the same, or none are right and I need to change them all).

If I was using a spreadsheet like Excel, that would take 2 seconds (drag down a range, control-d duplicate), but I haven't found a GUI SQL editor that can do this, so I click-copy-paste-click for 15 mins. This is time-wastey and feels silly. There's got to be an easier way.

(I realize that technically the app would have to do 60 individual updates, but that would be behind the scenes while I move on to more fun parts of my day, right?)

So is there a GUI SQL app that emulates the "copy down" of spreadsheets? Or if I were more clueful, is there a stored statement or "macro" I could make in one of these things that would take the selected field in the current record and dupe it down down down? If so, kick me in the right direction: I'm no database geek but I'm not wholly useless technically, either. I can probably learn from a few examples. (15 mins a day times 200 days a year is enough time to be worth reading a book!)

I'd prefer OSX but I would use Windows or Linux for a few minutes per day if the right tool is out there. For OSX, I've tried out the official mysql client, RazorSQL, Navicat, SequelPro, and SQLgrinder. Some are better than others at the copy-paste and ad hoc correcting I do, but none have "duplicate down".

These seems simple enough (to my pretty little head) that I must be missing something.

How to cross-ref two lists in Excel and remove duplicates?

GilloD — Fri, 14 Nov 2008 08:10:10 -0800

Excel-filter: I've got two sheets of data (Firstname, e-mail, lastname, customer #). I'd like to look for and remove any duplicate entries. More inside. I'm updating our customer e-mail newsletter mailing list. No one has done this in about a year. In that year our list of 20,000 names has dwindled to about 15,000. I have that list of 15,000 with bounces and unsubscribes removed. There are a few operations I want to perform to bring the list up to speed:

-People can opt-in to our newsletter when they buy something. This flags their customer entry in the shipping/inventory software but does NOT add them to the mailing list. I pulled a list of everyone who wants to get the e-mail from the last 6 months. I just want to cross-ref this list with the existing list AND with unsubscribes to make sure they didn't get in another way or have opted out since.

There are a couple of operations that happen in there. All of them come back to the same basic principal- To look at a sheet, or two sheets, and cross ref them looking for duplicates. If there's a duplicate, delete one of them.

This seems like it should be easy but I'm having a hell of a time. A Google search brings up lots of expensive add-ons but no real advice. Excel-ers of the world: HALP!

Yes, I know I sent that email four seconds ago...

freudenschade — Fri, 08 Aug 2008 09:33:59 -0800

Gmail IMAP + Thunderbird: I set up Thunderbird as my email client last week, and everything has been smooth sailing until today. When I opened T-bird, every email I've ever written was in my Inbox! Please make it stop! As of this morning, every email I send appears immediately in my Inbox (both in Thunderbird and the Gmail web interface). I created and ran a filter in T-bird that moved all emails "from" my address to "Gmail > Sent Mail" -- this moved all the old emails, but new ones keep appearing in the Inbox, and I'd really rather not have to run that filter all the time.

Notes about my setup:
- Google Apps Gmail
- Thunderbird 2.0.0.16
- Have IMAP for the same account set up in Outlook 2007, which I open occasionally to for email testing purposes
- Some filters set up within Gmail and T-bird, none of which have any settings regarding Sent Mail

I can provide other details as well if it would help. Any insight would be very much appreciated.

Streamline a conversation

colecovizion — Thu, 24 Apr 2008 14:49:54 -0800

Outlook Cleaning/Archive Gurus: Help me eliminate emails with similar (but not duplicate) content! I'm having a hard time finding a way to automate the search and deletion of emails with similar content. I have a manual process that I'm trying to automate.

The method: Every time an email is responded to, the full text is usually retained from the previous response.

Example (without attachments):

* Person #1 sends email
* Person #2 replies to email (includes the first email's full text)
* Person #1 replies to email (includes the first two emails)

Why not just keep the latest response in the conversation and any emails that have unique attachments? This way, you can reduce the Outlook size without compromising any data! Of course, this gets more complex with multiple recipients and responses.

The setup: Let's assume 1000 emails are in 1 giant archive folder. These emails include my sent email and my received email that are ready for archiving. Hence, my manual process....

* Sort by conversation (more often than not, the email text in the body is duplicated each time there is a response)
* Delete the oldest emails within a conversation UNLESS there is a unique attachment or unique text that isn't already in a newer email

In a world without attachments, this would usually keep the most current email in the conversation (which has all of my replies and their replies).

I know this is a very specific request, however, I think this method is pretty snazzy (to blow my own horn here) because it keeps all the unique information and it eliminates the duplicate information. Hence, smaller backups and better searches.

Any guidance is appreciated! Also if you have any Outlook email tricks please post for all to see!

Excel 2007 - Duplicate Data Removal Issue

Sandor Clegane — Wed, 16 Apr 2008 11:26:14 -0800

Excel 2007 - Is there a way to remove ONLY a duplicate row leaving the first unique entry? I have a list of sweepstakes entries that need to be pared down to only unique entries. When I attempt to remove duplicate data it usually removes both the duplicates AND the first entry. I need to find a simple way (without formulas, if possible) to separate the duplicates from the unique entries.

The data columns are as follows: Entry Date, First Name, Last Name, Email, Birthdate, Address, City, State, Zip, Sex, Phone Number, Opt In 1, Opt In 2.

I need to remove all duplicate email addresses from the list. There are a few issues with the way I currently do things. If I just highlight the email column and remove duplicate data it shifts all the unique addresses up so now the data is out of alignment. If I select the entire spreadsheet and remove duplicates it doesn't come up with the same number of duplicate entries as I get with just the email column. Finally, if I apply conditional formatting to the duplicate entries it highlights all versions of the same email address. Thus, I would be removing all entires for that person. I need to allow them one and remove the rest.

I realize there are ways to do this with formulas but I have to do this often and applying a formula to tens of thousands of rows is very, very cumbersome.

Oh Hive Mind, please help me!

Emergency duplicate Lexus key

cainiarb — Sun, 24 Feb 2008 16:38:11 -0800

Need a duplicate Lexus RX300 key I need to get a couple duplicate keys for my Lexus. I don't need one with all the bells and whistles to open/lock doors, etc. This is for an emergency key to be given to our daughter and one to be left at our weekend cottage in case I should lose mine. Is there anyway to get a couple emergency keys made without it costing $150 plus each?

Duplicate events created in GCal using Firefox 3 Beta 1?

electric_counterpoint — Wed, 28 Nov 2007 10:36:06 -0800

Why has Google Calendar started to duplicate my new events (same day, same time)? This only happens in Firefox 3 Beta 1. Obviously, I’m using a browser that Mozilla hasn't recommended for day-to-day use, so if it bothers me too much, I'll just go back to FF 2 for the time being.

But, has anyone else noticed this? Whenever I create a new event, it shows up twice, side-by-side. It only happens when I click on a day, drag out a duration, and enter in the details from there — it doesn't happen with the Quick Add field. It didn't happen with FF 2, and it still doesn't happen with Safari. Anyone found a fix?

Spring clean needed

mjlondon — Fri, 09 Mar 2007 08:14:07 -0800

Is there a program out there that will help me organise my hard drives? My primary hard drive is pretty much full and I am keen to move some content over to another drive. I know though that I can save alot more space by deleting duplicated and unused files. I've used CCleaner but that's pretty focused on deleting programs and temporary files. Is there anything that might identify mp3/jpeg/wmv etc I haven't accessed for some time or are duplicated elsewhere on the hard drive? Or would help streamline my folders? This particularly isn't helped by the fact that iTunes seems to keep reverting it's saved music folder so I have a few locations being accessed from there. I really dislike it being so untidy so any help would be great.

How do I remove duplicates and then merge two excel files?

perpetualstroll — Wed, 07 Feb 2007 18:25:32 -0800

In MS Excel, how do I remove duplicates, clean up a document and merge two files together? I have found a few threads in Google, but none of them have quite what I'm looking for, I'm hoping someone will have experience doing this. Right now, I have to do this manually, but I figure there has to be a more efficient way of completing this.

This is a bit complicated, but I'll try to simplify as best as I can:

I have two excel files, A and B.

File A contains the fields PART NUMBER and PRICE and is laid out as follows:

a1000 $1.25
a1000 $1.25
a1000 total $2.50
a1001 $3.25
a1001 total $3.25
a1002 $3.31
a1002 $3.31
a1002 total $6.62
etc. etc.

Questions:
1) How do I remove all the duplicate records?
2) How do I remove all the rows that contain the word "total" in the PART NUMBER field?

File B contains a master list of a lot more information for other parts and the formatting needs to be retained. After cleaning up File A, is there a import those values to a designated column for their corresponding part numbers in File B?

Any help would be MUCH appreciated.

Nuke Duplicate Firefox Extensions

SansPoint — Fri, 14 Jul 2006 13:32:40 -0800

How do I remove duplicate extensions from Firefox 1.5 Mac? I installed a couple extensions, FlashGot and Download Status Bar, but somehow, there's duplicate entries in the extensions window. I managed to remove the originals, but the dupes remain. Oddly, the dupes show the generic extension icon, and when I select them, the Uninstall button greys out. How can I get rid of them?

RSS Mashup & Duplicate Removal

will — Sat, 08 Apr 2006 20:01:33 -0800

Looking for a tool that can take multiple RSS feeds, strip duplicate entries and output a single feed. None of the RSS meshing tools I'm seeing seem to offer the duplicate removal. I'm subscribed to searches through 20+ feeds to make sure I catch all available references of a URL on blogs, courtesy of MonitorThis, but the number of duplicates is becoming a big problem.

iTunes file dupes and bad tag reads.

mykescipark — Sat, 18 Mar 2006 05:36:47 -0800

iTunes is creating duplicate copies of everything - and isn't faring too well with the id3 tags, either. I apologize in advance for yet another iTunes (PC) question.

I have my Windows iTunes (v6.0.1.3) set up to auto-organize my music into the standard iTunes folder configuration [iTunes Music > Artist > Album > Songs]. However, there seem to be a few bugs that I can't quite work out.

1. When using the "Add Folder to Library" function or when dragging files into iTunes, it creates TWO copies of each album in the iTunes directory. I then have to manually select one set and delete them. It isn't simply adding the files from the source directory as well - it's actually just double-importing them into the iTunes folders. (For what it's worth, it doesn't have this problem when I add music with the Add File(s) To Library function.)

2. Once imported, when I look in the Name (song title) field, it will list both "Artist - Song Title"instead of just "Song Title." The only way I can get it to list just the song title is by actually cueing up each song one by one, as if previewing every song in a given directory/album. I can't seem to find a universal command which will fix them in bulk when this happens.

For what it's worth, 95% of the albums in question were originally ripped in iTunes, so it's not like it has to figure out another program's id3-tags; I'm just reorganizing my file structure, and this is adding twice the necessary work.

Any thoughts? I could just tell iTunes to search for duplicates and nuke them all later on, but it seems like there should be a way to do this right the first time.

Excel bells & whistles

Amizu — Wed, 07 Dec 2005 14:28:58 -0800

Microsoft Excel. Is it possible to insert some "alarm" function that will notify me if I insert duplicate text in later cells? I am creating an Excel document for data entry. Multiple people will be entering the data from different cases. The first thing we will do is create a unique file name and type it in the first column. Then we will add all kinds of other data in adjoining columns. I would like to insert some feature that will alert the data-entry person if the same file name (first column) is entered later on, so that we don't accidentally enter data for the same case twice. Is this even possible in Excel?

Searching for duplicates in iTunes

DandyRandy — Wed, 29 Jun 2005 11:47:02 -0800

I wish the Find Duplicate Songs feature in iTunes was a little fuzzier in its matching capabilities. The way it works now is that the artist and the song title have to match exactly for it to count it as a match, which works well enough, but over the years I've had some files sneak in with (or without) a 'the' at the beginning of the song title. iTunes will not match those. The only way I see for scanning for these kind of duplicates is the old fashioned way of eyeball-comparing the 'the' files with the entire list. iTunes doesn't offer the option of opening a second window, and if I try to filter for 'The *' for an export it brings up all songs with 'the' anywhere within its title. How can I query for all songs that start with 'The'? Or is there a better way of making that match?

How can I get my girlfriend a key to my new apartment's front door ?

drezdn — Wed, 29 Sep 2004 20:56:17 -0800

Key Controversy... How can I get my girlfriend a key to my new apartment's front door or work out an arrangement so she can get in when she wants too? [mi] I just moved into a new building about a block away from my girlfriend's apartment. The building included a frontdoor key for both me and my roommate, but the key says do not duplicate.

Even if I can get a copy made some how, my girlfriend has already got on the nerves of the new building manager (apparently only tenants are supposed to let people into the building- but I sent her down to let in the cable guy). However, since I have a slightly nicer apartment, and fullblown awesome cable, she would like to hang out there. I'm worried that making a copy of the key will severly piss off my building manager.

Is there some clever work around that will allow me to either get a key made, or work out something so my girlfriend can come and go as she pleases?

Is there a way for the computer/a program to search for multiple duplicate files on a computer?

agregoli — Thu, 22 Jul 2004 19:50:36 -0800

Is there a way for the computer/a program to search for multiple duplicate files on a computer?

I also remember a program that used a pie chart to show you what is taking up the most space on the computer? What files and folders?

Help!