Home > How-To > Remove Duplicate Tracks in iTunes

Remove Duplicate Tracks in iTunes

November 14th, 2008

iTunes does a pretty good job keeping your music organized however when you have a ton of music and store it on a NAS for sharing, iTunes gets pissed when it can’t connect and will automatically change your library folder back to home/music.

I had this happen to me three times within the past month and as a result I have three times the music collection (from duplicates being copied without me noticing).  Here is a simple way to get your nice clean library back without having to buy anything.

First you need a copy of fdupes (Linux).  It can identify unique files and tee or > them to a list.  For that you would change to your music directory and run :

fdupes -rf . | tee duplicates

This process may take a while depending on how many files you are comparing. Next you would have to remove them however bash doesn’t like spaces in file names so lets toss this in a little script.

#!/bin/bash
IFS=$'\n'
for x in `cat duplicates`; do rm -v $x; done

And there you have it, no issues with spaces and a nice clean library although beware - you may want to run fdupes twice. Here’s why :

When you issue the -r flag it scans directories recursively.
The -f flag says “keep the first one you find from being piped to the duplicates list”

You should probably run -r and -rf (piped into two individual files) then use vimdiff to compare what you have against what you will potentially be deleting.  I had so much crap in there I just run -rf and deleted.

Happy organizing.

How-To , ,

  1. Jason
    November 14th, 2008 at 08:24 | #1

    Why do you not just open iTunes.

    File -> Show Duplicates.

  2. November 16th, 2008 at 11:21 | #2

    The answer is simple -iTunes “show duplicates” feature sucks for huge libraries.

    For example I have every our lady peace album as well as their Live recordings, when I use iTunes to show dupes I have to now compare against albums as well as song titles, I do not want to accidentally remove my live recordings by blindly selecting every other track.

    When you have > 5000 tracks in your library this gets EXTREMELY tedious and can take forever, plus humans are stupid and screw things up.

    Luckily fdupes generates a hash per binary and compares against all other hashes in the specified directory. This is far superior as it compares not only titles but the actual file content, so different recordings with the same titles aren’t added to delete list.

    With fdupes you pass the -f flag and it will keep the first occurrence of a duplicate hash which is so efficient, saves me from a heap of bash scripting.

    In total I had 15, 000 tracks in my library with only 5000 unique tracks, how long would it take a human to compare all of that data? Probably a week - I will stick with fdupes and trust my geek instincts.

  3. December 17th, 2008 at 17:32 | #3

    Действительно интерестно.

  4. December 19th, 2008 at 00:50 | #4

    Google says :
    Translation: Russian » English
    RU : Действительно интерестно.
    EN : Indeed smarter.
    Thanks!

  1. No trackbacks yet.