This software category is in need of an editor. If you would like to give something back to the freeware community by taking it over, check out this page for more details. You can then contact us from that page or by clicking here.
Introduction
If you are like me, you often find that you create duplicate files as you rename or use files in another context, or just backup more than you had intended. Duplicate File Detectors help you find and/or delete these duplicates and regain the storage space that they occupied.
Judging from some of the recent comments, it would appear that some cautions should be noted before using this type of software. One might also note that with the cost per gigabyte rapidly decreasing (As of Oct 2011, 500 GB drives can be found for $39), redundancy is no longer such a bad thing.
Don't ever just blindly run the program, find all the duplicates and then click "remove". This can be disastrous in the sense that even though filenames are the same, sometimes the content is different. For example, you may have multiple icons with the same name, but of differing sizes, e.g., 256x256, 128x128, etc. Or, you may have a photograph that has been modified when placed in a different folder but retains the same filename.
Don't scan too much at once. A complete scan of drive C would not be smart. However, scanning a few related folders at one time is much more efficient and the results less overwhelming.
Despite the best software, human intelligence may still have to be used. It is rumored that Abraham Lincoln stated that common sense was very uncommon. It is best, like eating an elephant, to take one bite/one small scan at a time.
OK, rant finished.
Discussion
The following is an example of why you may require duplicate file software: you have 50 duplicates and you need to decide what to do with each file, but what you get are 50 pop-ups, one for every file. It is for this reason that Duplicate Cleaner is my No.1 program. Duplicate Cleaner has a very streamlined user interface divided into tabs. The scanning job sequentially progresses to the next tab. It also scans for files matching their CRC checksums. Scanning is very fast in comparison to Clone Spy.
Scan results are shown as a list in the results tab, where you can select any file by looking at its name, size, date, location, or use ‘Selection Assistant’.
The Selection Assistant selects files by date, smaller/greater size, or by having a specific string in file name/file path. It offers image preview of common image formats, and has a music search mode which can involve bit rate and size. It can also create hard links of the deleted duplicate files to the single file which you opt to keep. Hard linked files behave like any standard files, but are cross-linked to one single file. Read further details about hard links before using this option.
NB: Duplicate File Cleaner has been updated to version 2, with many improvements though it now requires the Microsoft .NET Framework version 2+ to be installed. It remains my number one choice..
Anti-Twin This very powerful program performs well and adds the ability to search unicode files. It will indeed be the No. 1 choice for many users. It has many great features and has many helpful hints. It is speedy, efficient and the unicode capabilities are a definite plus.
CloneSpy is the one of the oldest programs in its class. It's certainly not the fastest, but it's definitely the smartest. You can specifically select what is to be scanned and this can include multiple drives, multiple folders, or just individual folders. It has the ability to detect duplicates by CRC, by file name, by CRC and file name, and by file name and size. It will also detect zero length files. The program does not need to be installed but can be run from the executable so it's a good candidate for your USB Flash drive toolkit. It also throws up duplicates in a way that at least makes you think about what you are deleting. Click-happy users may find this latter feature an annoyance; to me it's a safety feature.
Fast Duplicate File Finder is yet another option. It is fast and easy to use. It also features two specialty searches: Graphics and music. However, these functions are now limited in the free version.
Double Killer is yet another option. It is fast and easy to use and does a credible job and it has enough options to satisfy most users.
Exact Duplicate Finder is yet another option. It is fast and easy to use, does a credible job and it has enough options to satisfy most users.
AllDup is yet another good option. It is fast and easy to use, does a credible job and it has plenty of options.
Quick Selection Guide
Duplicate Cleaner
8
Gizmo's Freeware award as the best product in its class!
Runs as a stand-alone program on a user's computer
Fast scan, selection assistant, music search mode
Requires Microsoft .NET Framework version 2+ to be installed
A portable version of this product is available from the developer.
Windows 98/ME/2000/XP/Vista/7
Editor
This software category is in need of an editor. If you would like to give something back to the freeware community by taking it over, check out this page for more details. You can then contact us from that page or by clicking here
by Anonymous20120208 (not verified) on 8. February 2012 - 22:53(88542)
Bigbang Doublekiller portability.
One of its options lets user choose whether to save options to registry, so that's not "stealth". Otherwise, I don't know if it can run in stealth.
I'd call it portable.
You just unzip the download and run the exe
It is easily copied and run from any folder.
Latest freeware version is still 1.6.2.82 from 2007, so maybe I'll try another from this review.
Because of http://www.bigbangenterprises.de/en/doublekillerpro/comp.htm
"Maximal file size (single file) 2 GB"
You can use file search app to find larger files.
comp.htm page says Pro is faster, but free seems fast enough to me.
by Simon0362 (not verified) on 30. January 2012 - 9:48(88016)
Just finishing using Duplicate cleaner - which removed 9000 duplicates from my badly organised photograph archives. Brilliant job, thanks for the excellent review
The installer for CloneSpy 2-62 gives the option to "install" all files to a single directory, and not register the files with Windows. In this mode, it is truly "portable." The installer suggests that the files in the selected folder can be copied to other media/folders, and run from there. I have used this option successfully.
by KarenA (not verified) on 10. September 2011 - 13:17(79369)
I tried Duplicate Cleaner, but it seems to be dumbly coded. Even when exact binary match is specified, it seems to be evaluating MD5 for *ALL* files instead of just the ones with the same filesize, which means it's orders of magnitude slower than it need be.
by Kron Kyrios (not verified) on 12. June 2011 - 16:34(73681)
To the best of my ability to tell, Anti-Twin will not detect duplicate *.exe files. I have tried all sorts of things to get it to do so. It even hangs when searching after specifying exe's in the filter match.
My final troubleshooting attempt was to create a folder and subfolder within, and make one copy of an exe to each (no other files present). No matter what combination of switches I tried, it would not find them. When I renamed the extensions on these files, it finds them right away.
This may not be a problem for many people, but I specifically downloaded this to deduplicate my freeware collection.
If I am missing something, please let me know. If I am correct in my findings, let this be a notice to those who have similar needs.
"Duplicate Cleaner" seems to do the trick, just fine. Since Duplicate Cleaner appears to handle Unicode as of version 2.0, it satisfies another critical need for me.
by TubaMamma (not verified) on 27. May 2011 - 7:58(72763)
I have used Double Killer for a long time and like a lot of things about it, but as I have done a overzealous job of backups what I really want is to have the files compared, then a option to delete all dupes in a certain location. Thereby leaving behind only the ones that aren't in the desired location.
So what, if any, program can set to search C://users/xxx/pictures/vacation/1 and C://users/xxx/pictures/vacation/2 then let you delete all the dupes from vacation2. DoubleKiller lists option of first or last dupes but on large jobs the folder order moves, or I wanted different sets. I'd settle for ctl+click to select many at once rather than individual clicking 400 dupes.
by FreakOne (not verified) on 21. May 2011 - 17:33(72444)
One can unpack the compressed file and find duplicates.
But, none can pack files back into any archive type.
For the .zip format, this unpack-find-remove-repack method can be implemented.
by Fresh Choice (not verified) on 22. April 2011 - 9:23(70690)
I have found Exact Duplicate Finder the easier to use since it deletes by folders easily, while in the Duplicate Finder you have to setup the master folder and only one can be set.
I liked duplicate cleaner, but in the window of results, I can't see an option to choose a folder to delete, so you have to take out the dupe files one by one which is a shame as I have about 30,000 lol
Edit* AntiTwin is the same. I guess that means they all probably going to be?
This product works well and seems to best quite fast.
Will get up a review and rating in the near future.
This field is getting rather crowded with very, very good products!
by gggirlgeek (not verified) on 25. December 2010 - 11:51(63221)
Thanks, Timmi. I will try AllDup to see if it lives up to its description.
I don't care for Duplicate Cleaner. It doesn't have any way to exclude files or folders. You can include anything you want of course but when you have a few pesky filenames that keep coming up, and thousands of subfolders to clean this would seem like a basic requirement.
I might try it with music files, but even here a basic function is missing: the ability to filter within a certain size range. Name range would be handy too, come to think of it. For example, you have 12 versions of the same song, some by different artists, some live versions, etc. You would need to filter the title for phrases with, say, 3 matching words (to eliminate things like "live" and "feat, or messy tags.) Then you could do a second pass to filter for matches within, 50kb filesize, OR within 3 seconds Length (also missing.)
To do a really thorough job with a large music collection, though, this program is missing the essential properties: Bitrate and Track_Volume (replay gain.) The latter is even missing in most mp3 taggers though. It eliminates crappy sounding files converted to high bitrates as an attempt compensate. I have often been surprised when a 168kb song beats a 320kb copy once I listened to it.
by DV (not verified) on 4. January 2011 - 16:24(63796)
Duplicate Cleaner 2.0 has folder/file exclusion and size criteria. It also has some fuzzy processing on music tags. Currently in beta (see dv forums) but due to be released soon.
by mikeg (not verified) on 24. December 2010 - 4:27(63171)
Hard drive has 6 partitions. Through time I have copied the same program on different drives. These are the ones I would like to delete the duplicates. Will any on this list do that?
Have a separate downloads folder on D drive for downloaded programs. I try not to install anything directly on C drive unless I have to do so. If operating system crashes, I can reload it and not lose my downloaded programs. I may have to reinstall them, but the installation file is still present.
I have moved my document folder to F drive for same reason.
by Peter Wills (not verified) on 31. December 2010 - 1:03(63578)
If I am understanding what you are trying to do then DoubleKiller will do just that. I have been testing it myself over the past couple of days in a Network environment as it seems to be the only program of the ones listed here or others I could find that allows you to specify unlimited locations to search and provides separate exclusion masks by file type, file attributes and file size.
It also accepts UNC folder names whereas most of the other programs will only search actual drives, including mapped drives.
It also runs straight from the exe file without installation and allows you to specify whether to save options settings in the registry. It will also remove those registry settings if you wish.
I, personally, do not know of any programs that do what you are asking for. Does anyone out there know of a program that will do this?
I would note that a hard drive crash, as opposed to a OS crash will still obliterate many / most of your files.
For this reason, a second hard drive — either internal, if possible, or external — would increase your reliability but still make it difficult to check for dupes.
Nice review Dave. One thing I have found that separates CloneSpy from the rest is that it seems to be the only one that allows duplicate file checking between 2 separate directories (or groups of directories) rather than within the directories.
I had a situation where I had Directory A where a co-worker had inadvertently saved some files. They should have been saved in Directory B or in some subdirectory of Directory B. I had to find out which files in Dir A had duplicates somewhere within the Dir B structure (NOT if Dir B had duplicates within itself or Dir A had dups within itself.)
CloneSpy allows you to set up 2 pools of directories and the cross check between the pools for duplicates. All the others check for duplicates within these 2 directories but can't be set up to cross compare only.
I have to disagree with your point about CloneSpy because DoubleKiller will do exactly what you say CloneSpy is the only one listed here will do and DoubleKiller is not limited to comparing 2 "pools" of folders. You can specify unlimited folders in different drives and partitions and DoubleKiller will find every duplicate instance across ALL specified locations.
You can also use UNC names to specify locations rather than having to rely on the location you want to search being a local drive or a mapped drive. None of the others listed here or others that I have found online will allow you to do this.
I have tested DoubleKiller in a real networked work environment and it found every instance of duplicate files in multiple locations across any network locations I specified.
Subscribe to our RSS feed or enter your email address below to get the feed delivered to you by email:
We are currently looking for people with skills and/or interest in the following areas:
Go straight to the Quick Selection Guide
Printer-friendly version
Comments
Bigbang Doublekiller portability.
One of its options lets user choose whether to save options to registry, so that's not "stealth". Otherwise, I don't know if it can run in stealth.
I'd call it portable.
You just unzip the download and run the exe
It is easily copied and run from any folder.
Latest freeware version is still 1.6.2.82 from 2007, so maybe I'll try another from this review.
Because of http://www.bigbangenterprises.de/en/doublekillerpro/comp.htm
"Maximal file size (single file) 2 GB"
You can use file search app to find larger files.
comp.htm page says Pro is faster, but free seems fast enough to me.
Just finishing using Duplicate cleaner - which removed 9000 duplicates from my badly organised photograph archives. Brilliant job, thanks for the excellent review
thanks for listing those software
The installer for CloneSpy 2-62 gives the option to "install" all files to a single directory, and not register the files with Windows. In this mode, it is truly "portable." The installer suggests that the files in the selected folder can be copied to other media/folders, and run from there. I have used this option successfully.
Just tried the latest Anti-Twin. It couldn't finish comparing the contents of 2 USB drives (500 GB, 2 TB). So I wouldn't recommend Anti-Twin at all.
I like Duplicate Cleaner, but it crashes when building file list. Very discouraging.
I tried Duplicate Cleaner, but it seems to be dumbly coded. Even when exact binary match is specified, it seems to be evaluating MD5 for *ALL* files instead of just the ones with the same filesize, which means it's orders of magnitude slower than it need be.
That would be dumb, but it's not actually true. Hashes are calculated and byte matches performed for equal size files only.
Hello,
Can you please add as a CON to the "Aulogic" software?
It's the fact that it can't look for duplicates in Network Drives. Even if they are Mapped as local drives.
Thanks.
To the best of my ability to tell, Anti-Twin will not detect duplicate *.exe files. I have tried all sorts of things to get it to do so. It even hangs when searching after specifying exe's in the filter match.
My final troubleshooting attempt was to create a folder and subfolder within, and make one copy of an exe to each (no other files present). No matter what combination of switches I tried, it would not find them. When I renamed the extensions on these files, it finds them right away.
This may not be a problem for many people, but I specifically downloaded this to deduplicate my freeware collection.
If I am missing something, please let me know. If I am correct in my findings, let this be a notice to those who have similar needs.
"Duplicate Cleaner" seems to do the trick, just fine. Since Duplicate Cleaner appears to handle Unicode as of version 2.0, it satisfies another critical need for me.
Click File Filter
then delete the list of "additionally by default NOT" extensions
It has always worked for me
Despite its classic interface, Duplicate cleaner is awesome. +1 for your top pick!
I have used Double Killer for a long time and like a lot of things about it, but as I have done a overzealous job of backups what I really want is to have the files compared, then a option to delete all dupes in a certain location. Thereby leaving behind only the ones that aren't in the desired location.
So what, if any, program can set to search C://users/xxx/pictures/vacation/1 and C://users/xxx/pictures/vacation/2 then let you delete all the dupes from vacation2. DoubleKiller lists option of first or last dupes but on large jobs the folder order moves, or I wanted different sets. I'd settle for ctl+click to select many at once rather than individual clicking 400 dupes.
Exact Duplicate Finder will do this for you :)
I have found a flaw that can lead to data loss in duplicate cleaners.
It's nothing huge but Exact Duplicate Finder will have it fixed soon.
All these utilities lack something very important: the ability to look within compressed files.
One can unpack the compressed file and find duplicates.
But, none can pack files back into any archive type.
For the .zip format, this unpack-find-remove-repack method can be implemented.
I have found Exact Duplicate Finder the easier to use since it deletes by folders easily, while in the Duplicate Finder you have to setup the master folder and only one can be set.
Best is Duplicate Cleaner( menu+options ),but to meni deleting errors!
Duplicate Cleaner + Auslogics Duplicate File Finder = ;)
I liked duplicate cleaner, but in the window of results, I can't see an option to choose a folder to delete, so you have to take out the dupe files one by one which is a shame as I have about 30,000 lol
Edit* AntiTwin is the same. I guess that means they all probably going to be?
Just right click on a file and do 'mark all by folder', or use the selection assistant.
Thanks for the list! I'm going with Anti-Twin; seems to have everything that I need (Interface, ease, file selection/exempt, music)
Thanks Again!
Greased Peace in the middle east!
Hi,
I found another tool that looks pretty robust as a duplicate file remover. Here it is: http://sourceforge.net/projects/yadfr/
Thanks for the find.
This product works well and seems to best quite fast.
Will get up a review and rating in the near future.
This field is getting rather crowded with very, very good products!
Thanks, Timmi. I will try AllDup to see if it lives up to its description.
I don't care for Duplicate Cleaner. It doesn't have any way to exclude files or folders. You can include anything you want of course but when you have a few pesky filenames that keep coming up, and thousands of subfolders to clean this would seem like a basic requirement.
I might try it with music files, but even here a basic function is missing: the ability to filter within a certain size range. Name range would be handy too, come to think of it. For example, you have 12 versions of the same song, some by different artists, some live versions, etc. You would need to filter the title for phrases with, say, 3 matching words (to eliminate things like "live" and "feat, or messy tags.) Then you could do a second pass to filter for matches within, 50kb filesize, OR within 3 seconds Length (also missing.)
To do a really thorough job with a large music collection, though, this program is missing the essential properties: Bitrate and Track_Volume (replay gain.) The latter is even missing in most mp3 taggers though. It eliminates crappy sounding files converted to high bitrates as an attempt compensate. I have often been surprised when a 168kb song beats a 320kb copy once I listened to it.
Duplicate Cleaner 2.0 has folder/file exclusion and size criteria. It also has some fuzzy processing on music tags. Currently in beta (see dv forums) but due to be released soon.
Hard drive has 6 partitions. Through time I have copied the same program on different drives. These are the ones I would like to delete the duplicates. Will any on this list do that?
Have a separate downloads folder on D drive for downloaded programs. I try not to install anything directly on C drive unless I have to do so. If operating system crashes, I can reload it and not lose my downloaded programs. I may have to reinstall them, but the installation file is still present.
I have moved my document folder to F drive for same reason.
If I am understanding what you are trying to do then DoubleKiller will do just that. I have been testing it myself over the past couple of days in a Network environment as it seems to be the only program of the ones listed here or others I could find that allows you to specify unlimited locations to search and provides separate exclusion masks by file type, file attributes and file size.
It also accepts UNC folder names whereas most of the other programs will only search actual drives, including mapped drives.
It also runs straight from the exe file without installation and allows you to specify whether to save options settings in the registry. It will also remove those registry settings if you wish.
It is by far the best one I have found to date.
I, personally, do not know of any programs that do what you are asking for. Does anyone out there know of a program that will do this?
I would note that a hard drive crash, as opposed to a OS crash will still obliterate many / most of your files.
For this reason, a second hard drive — either internal, if possible, or external — would increase your reliability but still make it difficult to check for dupes.
Nice review Dave. One thing I have found that separates CloneSpy from the rest is that it seems to be the only one that allows duplicate file checking between 2 separate directories (or groups of directories) rather than within the directories.
I had a situation where I had Directory A where a co-worker had inadvertently saved some files. They should have been saved in Directory B or in some subdirectory of Directory B. I had to find out which files in Dir A had duplicates somewhere within the Dir B structure (NOT if Dir B had duplicates within itself or Dir A had dups within itself.)
CloneSpy allows you to set up 2 pools of directories and the cross check between the pools for duplicates. All the others check for duplicates within these 2 directories but can't be set up to cross compare only.
I have to disagree with your point about CloneSpy because DoubleKiller will do exactly what you say CloneSpy is the only one listed here will do and DoubleKiller is not limited to comparing 2 "pools" of folders. You can specify unlimited folders in different drives and partitions and DoubleKiller will find every duplicate instance across ALL specified locations.
You can also use UNC names to specify locations rather than having to rely on the location you want to search being a local drive or a mapped drive. None of the others listed here or others that I have found online will allow you to do this.
I have tested DoubleKiller in a real networked work environment and it found every instance of duplicate files in multiple locations across any network locations I specified.
Post new comment