We are looking for people with skills or interest in the following:
- Mobile Platform Reviews
- Rootkit Scanner and Remover
- Streaming Media Recorder
- Email Client
- Archive Manager Interested?Click here
This software category is in need of an editor. If you would like to give something back to the freeware community by taking it over, check out this page for more details. You can then contact us from that page or by clicking here.
If you are like me, you often find that you create duplicate files as you rename or use files in another context, or just backup more than you had intended. Duplicate File Detectors help you find and/or delete these duplicates and regain the storage space they previously occupied.
Judging from some of the recent comments, it would appear that some cautions should be noted before using this type of software. One might also note that with the cost per gigabyte rapidly decreasing (As of Oct 2011, 500 GB drives can be found for $39), redundancy is no longer such a bad thing.
Don't ever just blindly run the program, find all the duplicates and then click "remove". This can be disastrous in the sense that even though filenames are the same, sometimes the content is different. For example, you may have multiple icons with the same name, but of differing sizes, e.g., 256x256, 128x128, etc. Or, you may have a photograph that has been modified when placed in a different folder but retains the same filename.
Don't scan too much at once. A complete scan of drive C: would not be smart. However, scanning a few related folders at one time is much more efficient and the results less overwhelming.
Despite the best software, human intelligence may still have to be used. It is rumored that Abraham Lincoln stated that common sense was very uncommon. It is best, like eating an elephant, to take one bite/one small scan at a time.
OK, rant finished.
Discussion
Over the past few days, I've had the opportunity to take a look at a program by the name of SearchMyFiles by NirSoft. It didn't take long to realize just how well this program works. It features a clean and simple layout and the ability to perform both standard searches or duplicate file searches with as little or as many filters as you want. For standard searches, the plentiful search filters allow for some effective searching. However, I didn't find the filters to be as useful when searching for duplicate files though. Despite that, I was still impresses with the numerous search filters such as searching for files made within specific time frames, a certain file size range, certain file extensions, and so much more. Add on to this that the program can be used on of a USB drive, and you have a program that is not only useful, but flexible too. The only negative I could muster up on this one was that the search results are in a seperate window. Having to switch back and forth between search options and search results was a little bit of a drag. Although not a big deal, it would be nice to have everything in the same window. In my opinion, NirSoft has really hit the nail on the head with this one. I've tried the other duplicate file detector programs here and I can honestly say that this one is the best one here thus far.
Thanks to gzmhanu for suggesting that I review SearchMyFiles by NirSoft.
The following is an example of why you may require duplicate file software: you have 50 duplicates and you need to decide what to do with each file, but what you get are 50 pop-ups, one for every file. Duplicate Cleaner Free has a very streamlined user interface divided into tabs. The scanning job sequentially progresses to the next tab. It also scans for files matching their CRC checksums. Scanning is very fast in comparison to Clone Spy.
Scan results are shown as a list in the results tab, where you can select any file by looking at its name, size, date, location, or use ‘Selection Assistant’.
The Selection Assistant selects files by date, smaller/greater size, or by having a specific string in file name/file path. It offers image preview of common image formats, and has a music search mode which can involve bit rate and size. It can also create hard links of the deleted duplicate files to the single file which you opt to keep. Hard linked files behave like any standard files, but are cross-linked to one single file. Read further details about hard links before using this option.
Anti-Twin This very powerful program performs well and adds the ability to search unicode files. It will indeed be the No. 1 choice for many users. It has many great features and has many helpful hints. It is speedy, efficient and the unicode capabilities are a definite plus.
CloneSpy is the one of the oldest programs in its class. It's certainly not the fastest, but it's definitely the smartest. You can specifically select what is to be scanned and this can include multiple drives, multiple folders, or just individual folders. It has the ability to detect duplicates by CRC, by file name, by CRC and file name, and by file name and size. It will also detect zero length files. The program does not need to be installed but can be run from the executable so it's a good candidate for your USB Flash drive toolkit. It also throws up duplicates in a way that at least makes you think about what you are deleting. Click-happy users may find this latter feature an annoyance; to me it's a safety feature.
Fast Duplicate File Finder is yet another option. It is fast and easy to use. It also features two specialty searches: Graphics and music. However, these functions are now limited in the free version.
Exact Duplicate Finder is yet another option. It is fast and easy to use, does a credible job and it has enough options to satisfy most users.
Double Killer is yet another option. It is fast and easy to use and does a credible job and it has enough options to satisfy most users.
AllDup is yet another good option. It is fast and easy to use, does a credible job and it has plenty of options.
Quick Selection Guide
SearchMyFiles
9
Gizmo's Freeware award as the best product in its class!
Runs as a stand-alone program on a user's computer
Easy to use, plenty of search options, searches run quickly
A portable version of this product is available from the developer.
Windows 98/ME/2000/XP/Vista/7
Editor
This software category is in need of an editor. If you would like to give something back to the freeware community by taking it over, check out this page for more details. You can then contact us from that page or by clicking here
I think there is an important item being overlooked in the comparision.
Won't affect the selection of Best in Class though.
I'm using Windows 7 64bit, and NAS4Free on a Microserver on the home network. I have a lot of files strewn across USB hard drive. A recent external drive failure caused me to get a NAS solution.
So this duplicate file finder mechanism is really important for home use with the amount of digital information normal families accumulate.
Well, neither "Duplicate Cleaner Free" or "Auslogics Duplicate File Finder" will touch the windows share from my NAS server.
"SearchMyFiles" performs flawlessly.... Love it.
So could TSA add network share capable as part of it's tests please.
Duplicate Cleaner is clean and doesn't include a Bing Bar (or any toolbar) in the installer - you need to update your listing! My vote for the best though, as it has more features than most of the paid guys.
Thanks for your input. I'm still running through all the duplicate file detectors in this list. I will be sure to update all Duplicate Cleaner's particulars as soon as I get done testing.
by asaens (not verified) on 28. September 2012 - 14:18(99909)
Greetings, I'm trying to find a program similar to these that will allow me to search for 2 or more occurrences of files by name and also have the ability display the CRC-32, MD5, SHA1 or some other hash value of the files in order to locate some possible corrupted files. I'll assume one of the files is possibly corrupted and that further investigation may be needed if the hash values aren't the same. Most of these attempt to locate multiple occurrences by compare hash values and then displaying the ones that are exactly the same so my needs are a little different. I have used WinRar to do a recursive search from the root directory of a drive but, as you can imagine, more work is required to visually filter out the files that are listed only once. Any help would be greatly appreciated.
Another vote here for Visipics, specifically for hunting down duplicate piccies on drives...
It has a slider for adjusting the level of similarity, it will show more than just pairs (e.g. if it comes across three or more pics the same or similar), it has a decent preview pane, and most usefully it also lets you save the results to load and view later. Essential after it's spent ages searching a 500Gb USB drive and you don't feel inclined to deal with the results right then. Various other useful options too, it's my favourite so far for picture sorting.
Alternative to consider: dupeGuru (http://www.hardcoded.net/dupeguru/)
It even exists in a Music Edition and a Picture Edition.
(Note: The author calls it 'fairware', but in case of dupeGuru, in fact it's freeware)
by MClark (not verified) on 31. August 2012 - 16:28(98576)
I am searching for duplicate finder but the one I have downloaded and tested Fast Duplicate Finder. It worked well and found many duplicate however I still had duplicates in the "non-duplicates" file. I really would like something (I think mentioned above) that does a complete file scan to determine exact duplicates. I need it for files, images, and music. I have experienced a hard drive crash with replacement but still getting an incredible # of disk error events; so, I have files all over the place that have been copied back and forth to different places and a simple move to one folder does not do it--some have been renamed--they are all a mess. If anyone has a suggestion free preferred, but willing to purchase the best or tried and true.
Please appreciate that this is a freeware site, and therefore only freeware will be discussed here. Any postings about commercial software will be deleted.
by Pattersonsqld (not verified) on 7. July 2012 - 23:32(95874)
I tried Duplicate Cleaner Free but found a: it has limitations if using the free version & b: it was going to delete a 10MB version and suggesting keeping a 3MB version. NOT what I wanted to do & the absence of a help facility (just said couldn't connect & nothing in the text for anything in help & nothing online I could find.
Auslogics Duplicate File Finder was easy, clear in it's options & only compared EXACT matches which, face it, is what we really want in this situation. I don't want to keep a 12KB bit rate & delete the 320KB copy!
You can select to delete the duplicates to the Recycle Bin, Rescue Centre or delete permanently.
And it does it for free.
Didn't try the others but Auslogics did it for me.
by stxnpx (not verified) on 21. July 2012 - 17:29(96492)
I've used Duplicate File Finder and it worked really well. My favorite feature was the protective inability to delete every copy of a particular file. It will group duplicates together and you can either automatically or manually select which copies to delete, but the program will not force you to leave one copy/version of each file. When going through thousands of files and madly clicking away in boredom and frustration, that saved me several times! I just wish it was available for Macs.
by Jenna (not verified) on 14. June 2012 - 18:34(94873)
Hi I am a Mac user and am looking for a free duplicate file finder for my Umpteenthousand pics my start-up disk is very full but I don't want to get rid of them and don't want to back up more then one of the same thing I read [commercial reference removed]. I don't have windows at all. also I am running on OS X Tiger with a powerPC G4 Processor so I run into alto of problems with programs being archived. Can you help me?
I'm not too familiar with Macs myself. Try the forums on this website and someone would be more than happy to guide you in the right direction. Thank you for your question and for visiting this site! Good luck!
In this example, "B" folder has 2 duplicate sub-folders (Nevermind and In Utero) so they will be removed.
NOTE :
- It DOESN'T matter if the files inside are different, it only checks sub-folder names NOT files.
- Folders and sub-folders with different names (not existing in "A") won't be touched inside "B"
B
|____Nirvana
| |_____1992 - Incesticide <= V2
I then only have to copy my fresh files in "B"
I've tried so many different softwares but they won't do exactly the same thing...
If your "different" bitrate files (in folder B) have the same exact names as your "V0" bitrate files (in folder A), couldn't you just copy folder A on top of folder B? In that case, your "different" files (with the exact same name) would be automatically deleted and replaced by the "V0" bitrate files (of the exact same name).
by K8J6riYkNs (not verified) on 15. May 2012 - 19:41(93509)
I have a lot of audio and video files that, I'm sure, have become duplicated and I'm looking for a program that will simplify the task of cleaning these files. none of the programs listed seem to specifically be able to do this. Are there any recommendations for a program that will help me?
by John Lee (not verified) on 27. May 2012 - 10:10(94030)
Duplicate Cleaner Free has and audio mode which scans the tags, and so can help identify copies songs which may be different bit rates but have the same artist/track name.
by Milos (not verified) on 15. May 2012 - 5:42(93472)
I find that the best way to look for duplicate files is to look for files of the same length as even files of different sizes, names, bitrates etc can be duplicates. But because I haven't been able to find anything to do thi it can be a slow process. Does anyone know of a prog/app that searches for duplicated by the length (playing time) of the file? Many thanks.
by TC (not verified) on 4. September 2012 - 15:23(98758)
If you're wanting to manage a lot of audio / video files I would strongly suggest MediaMonkey. It has lots of options and abilities for finding duplicates, copying and pasting tags, etc. It's obviously no use for non-media files (or images) but might help you.
by jphotonica (not verified) on 10. May 2012 - 9:59(93300)
I'm desperately looking for something that will allow to choose a base folder and only find duplicate files based on that folder. In other words, I want to search all my external hard drives for any possible duplicates only based from one chosen folder on my D: drive partition where I keep my JPEG images. Ideally, I'd like to import the original .NEF image file versions of my D: drive JPEG folder images. These software that allows to choose base search folders will simply search all files to find all duplicates in all chosen folders. Anyone have any suggestions?
by Howardec (not verified) on 8. May 2012 - 17:44(93231)
First off, thank you for this web site, it has given me a lot of useful info on determining what I need. Basically I am looking for a program to find files with the same amount of specified characters in the beginning of a file name. In my folder I may have 10,000 files & all I am trying to do is find any files that contain the first 6 (or 8 or 10 depending, hopefully customizable)of the same characters. Like, in this directory I will have 3 files named Howard34.pdf, Howard38764.pdf & Howardplease.pdf, so when I search I want it to tell me that based on the first 6 characters, these files are a match. Also, if I change it to the first 7, then only the first 2 files will come up in the list because the first 7 characters are Howard3 & the third file is Howardp. Now I may be looking for the wrongly named program, what I am calling duplicates are only so by name & not size/tag info. I've read through "most" of the comments, ones I thought would pertain to my question & I've even installed a few programs (DupKiller being the closest so far) to try to do what I need. DupKiller allows a percentage of the file name, but not an exact number. Would any of these programs do what I need?
Thanks for all the help!!!
Howard
Subscribe to our Top Picks RSS Feed or enter your email address below to get the feed delivered to you by email:
We are looking for people with skills or interest in the following:
Go straight to the Quick Selection Guide
Over the past few days, I've had the opportunity to take a look at a program by the name of SearchMyFiles by NirSoft. It didn't take long to realize just how well this program works. It features a clean and simple layout and the ability to perform both standard searches or duplicate file searches with as little or as many filters as you want. For standard searches, the plentiful search filters allow for some effective searching. However, I didn't find the filters to be as useful when searching for duplicate files though. Despite that, I was still impresses with the numerous search filters such as searching for files made within specific time frames, a certain file size range, certain file extensions, and so much more. Add on to this that the program can be used on of a USB drive, and you have a program that is not only useful, but flexible too. The only negative I could muster up on this one was that the search results are in a seperate window. Having to switch back and forth between search options and search results was a little bit of a drag. Although not a big deal, it would be nice to have everything in the same window. In my opinion, NirSoft has really hit the nail on the head with this one. I've tried the other duplicate file detector programs here and I can honestly say that this one is the best one here thus far.
Printer-friendly version
Comments
I think there is an important item being overlooked in the comparision.
Won't affect the selection of Best in Class though.
I'm using Windows 7 64bit, and NAS4Free on a Microserver on the home network. I have a lot of files strewn across USB hard drive. A recent external drive failure caused me to get a NAS solution.
So this duplicate file finder mechanism is really important for home use with the amount of digital information normal families accumulate.
Well, neither "Duplicate Cleaner Free" or "Auslogics Duplicate File Finder" will touch the windows share from my NAS server.
"SearchMyFiles" performs flawlessly.... Love it.
So could TSA add network share capable as part of it's tests please.
Thanks for the awesome work, and awesome site.
Michael
Can any of these applications look inside compressed files (rar, zip, cab...)?
Testing is almost complete. Will have answers for you shortly.
Duplicate Cleaner is clean and doesn't include a Bing Bar (or any toolbar) in the installer - you need to update your listing! My vote for the best though, as it has more features than most of the paid guys.
Thanks for your input. I'm still running through all the duplicate file detectors in this list. I will be sure to update all Duplicate Cleaner's particulars as soon as I get done testing.
Greetings, I'm trying to find a program similar to these that will allow me to search for 2 or more occurrences of files by name and also have the ability display the CRC-32, MD5, SHA1 or some other hash value of the files in order to locate some possible corrupted files. I'll assume one of the files is possibly corrupted and that further investigation may be needed if the hash values aren't the same. Most of these attempt to locate multiple occurrences by compare hash values and then displaying the ones that are exactly the same so my needs are a little different. I have used WinRar to do a recursive search from the root directory of a drive but, as you can imagine, more work is required to visually filter out the files that are listed only once. Any help would be greatly appreciated.
Another vote here for Visipics, specifically for hunting down duplicate piccies on drives...
It has a slider for adjusting the level of similarity, it will show more than just pairs (e.g. if it comes across three or more pics the same or similar), it has a decent preview pane, and most usefully it also lets you save the results to load and view later. Essential after it's spent ages searching a 500Gb USB drive and you don't feel inclined to deal with the results right then. Various other useful options too, it's my favourite so far for picture sorting.
Great article by the way, thanks.
Alternative to consider: dupeGuru (http://www.hardcoded.net/dupeguru/)
It even exists in a Music Edition and a Picture Edition.
(Note: The author calls it 'fairware', but in case of dupeGuru, in fact it's freeware)
I am searching for duplicate finder but the one I have downloaded and tested Fast Duplicate Finder. It worked well and found many duplicate however I still had duplicates in the "non-duplicates" file. I really would like something (I think mentioned above) that does a complete file scan to determine exact duplicates. I need it for files, images, and music. I have experienced a hard drive crash with replacement but still getting an incredible # of disk error events; so, I have files all over the place that have been copied back and forth to different places and a simple move to one folder does not do it--some have been renamed--they are all a mess. If anyone has a suggestion free preferred, but willing to purchase the best or tried and true.
Please appreciate that this is a freeware site, and therefore only freeware will be discussed here. Any postings about commercial software will be deleted.
I tried Duplicate Cleaner Free but found a: it has limitations if using the free version & b: it was going to delete a 10MB version and suggesting keeping a 3MB version. NOT what I wanted to do & the absence of a help facility (just said couldn't connect & nothing in the text for anything in help & nothing online I could find.
Auslogics Duplicate File Finder was easy, clear in it's options & only compared EXACT matches which, face it, is what we really want in this situation. I don't want to keep a 12KB bit rate & delete the 320KB copy!
You can select to delete the duplicates to the Recycle Bin, Rescue Centre or delete permanently.
And it does it for free.
Didn't try the others but Auslogics did it for me.
Hey
AllDup works so far. I had to filter around 500000 files with different criteria.
But how do you get rid of duplicats with nirsoft ? There is no delete function!
Has anybody tried Duplicate File Finder from http://www.softpedia.com/developer/12oClocker-Software-15764.html
I've used Duplicate File Finder and it worked really well. My favorite feature was the protective inability to delete every copy of a particular file. It will group duplicates together and you can either automatically or manually select which copies to delete, but the program will not force you to leave one copy/version of each file. When going through thousands of files and madly clicking away in boredom and frustration, that saved me several times! I just wish it was available for Macs.
Duplicate Cleaner Free the best. 10/10.
Fantastically written article, I have to say. Thanks and best luck to you.
Hi Naravive,
Thank you for your comments! They are greatly appreciated! :)
Hi I am a Mac user and am looking for a free duplicate file finder for my Umpteenthousand pics my start-up disk is very full but I don't want to get rid of them and don't want to back up more then one of the same thing I read [commercial reference removed]. I don't have windows at all. also I am running on OS X Tiger with a powerPC G4 Processor so I run into alto of problems with programs being archived. Can you help me?
Hi Jenna,
I'm not too familiar with Macs myself. Try the forums on this website and someone would be more than happy to guide you in the right direction. Thank you for your question and for visiting this site! Good luck!
I'm looking for a tool to compare folders and remove duplicate folders (same name).
I have a folder (A) of mp3 with V0 bitrates that I want to use to replace my existing mp3 folder (B) made by different bitrates.
A
|____Nirvana
| |_____1991 - Nevermind <= V0
| |
| |_____1993 - In Utero <= V0
B
|____Nirvana
| |_____1991 - Nevermind <= V0
| |
| |_____1993 - In Utero <= 320
| |
| |_____1992 - Incesticide <= V2
In this example, "B" folder has 2 duplicate sub-folders (Nevermind and In Utero) so they will be removed.
NOTE :
- It DOESN'T matter if the files inside are different, it only checks sub-folder names NOT files.
- Folders and sub-folders with different names (not existing in "A") won't be touched inside "B"
B
|____Nirvana
| |_____1992 - Incesticide <= V2
I then only have to copy my fresh files in "B"
I've tried so many different softwares but they won't do exactly the same thing...
So do you know a way to achieve this?
If your "different" bitrate files (in folder B) have the same exact names as your "V0" bitrate files (in folder A), couldn't you just copy folder A on top of folder B? In that case, your "different" files (with the exact same name) would be automatically deleted and replaced by the "V0" bitrate files (of the exact same name).
I see what you mean, but it's a bit tricky to proceed that way.
Anyway, someone found a solution : http://www.donationcoder.com/forum/index.php?topic=31272.msg290737#msg290737
I have a lot of audio and video files that, I'm sure, have become duplicated and I'm looking for a program that will simplify the task of cleaning these files. none of the programs listed seem to specifically be able to do this. Are there any recommendations for a program that will help me?
Duplicate Cleaner Free has and audio mode which scans the tags, and so can help identify copies songs which may be different bit rates but have the same artist/track name.
I find that the best way to look for duplicate files is to look for files of the same length as even files of different sizes, names, bitrates etc can be duplicates. But because I haven't been able to find anything to do thi it can be a slow process. Does anyone know of a prog/app that searches for duplicated by the length (playing time) of the file? Many thanks.
If you're wanting to manage a lot of audio / video files I would strongly suggest MediaMonkey. It has lots of options and abilities for finding duplicates, copying and pasting tags, etc. It's obviously no use for non-media files (or images) but might help you.
I'm desperately looking for something that will allow to choose a base folder and only find duplicate files based on that folder. In other words, I want to search all my external hard drives for any possible duplicates only based from one chosen folder on my D: drive partition where I keep my JPEG images. Ideally, I'd like to import the original .NEF image file versions of my D: drive JPEG folder images. These software that allows to choose base search folders will simply search all files to find all duplicates in all chosen folders. Anyone have any suggestions?
Visipics has been great for my photographs.
+1 for VisiPics
First off, thank you for this web site, it has given me a lot of useful info on determining what I need. Basically I am looking for a program to find files with the same amount of specified characters in the beginning of a file name. In my folder I may have 10,000 files & all I am trying to do is find any files that contain the first 6 (or 8 or 10 depending, hopefully customizable)of the same characters. Like, in this directory I will have 3 files named Howard34.pdf, Howard38764.pdf & Howardplease.pdf, so when I search I want it to tell me that based on the first 6 characters, these files are a match. Also, if I change it to the first 7, then only the first 2 files will come up in the list because the first 7 characters are Howard3 & the third file is Howardp. Now I may be looking for the wrongly named program, what I am calling duplicates are only so by name & not size/tag info. I've read through "most" of the comments, ones I thought would pertain to my question & I've even installed a few programs (DupKiller being the closest so far) to try to do what I need. DupKiller allows a percentage of the file name, but not an exact number. Would any of these programs do what I need?
Thanks for all the help!!!
Howard