Gizmos Freeware Reviews  

Go Back   Gizmo's Freeware Forum > Freeware Forum > I Want a Freeware Program that ...

Thread Tools Display Modes
Old 14. Jul 2014, 07:02 PM   #1 (permalink)
Join Date: Aug 2008
Posts: 3
Default Duplicate finder feeding backup

I need software that will scan my HDD for duplicates and log the results.
From there backup all except the operating system, while backing up only one copy of any duplicate files.

The result is full backup of my HDD onto a USB HDD mirror image. Files that are not backed up due to duplicate status, will have a marker left showing where the duplicate file is supposed to be.

To restore: Re-install WindowsXP Pro SP III. From the free software, select "restore". Software checks for sufficient space on new HDD. If OK, restores all files and folders.

Periodic (daily, weekly, monthly) backups would only back up files that have changed since last backup. I'm no programmer, but it would seem all the elements are available as open source. Stringing the two processes, (dupe finder, backup/restore)would be extremely useful.
cebtoo is offline   Reply With Quote
Old 15. Jul 2014, 11:52 AM   #2 (permalink)
Senior Member
Join Date: Jul 2010
Location: New Zealand
Posts: 1,741

I don't think that all the elements are available in Open Source in the way you want them but there is a lot of similar stuff and there is at least one free product that does something similar just not on Windows.

There are many backup and archiving programs that try not to store duplicate data. They usually try to avoid storing duplicated blocks of data which are not necessarily files. This is what happens in .ZIP and .ARC files. There is possibly one that compares files to see if they are duplicated. I don't know if any generate anything like a log of duplicated files.

What you want is Data de-duplication. Backup programs that do this usually cost $$$ because they are used by enterprise-sized systems not personal users. One option is OpenDedup but it doesn't have a Windows solution. You could probably do it by booting Linux or using a NAS as a file server.

Here's some links to relevant topics in Wikipedia:
  • Backup
  • ZFS file system has deduplication and a lot of useful data storage features built into the file system but it is a memory hog because of all the file hash data that is needed to operate it. It's usually cheaper to buy disk space than memory. You can get ZFS as part of free NAS software (e.g. FreeNAS), Linux, OpenBSD and other operating system software.
  • rsync . There is a lot of free rsync software and some of the variations on rsync use file deltas between incremental backups. That means only the specific differences are stored for the same file in two backups. The first backup will have the file then the second backup only has each file's delta (differences). It's slightly possible that a product of this type has deduplication but I think I would have heard of it.
Better to light a candle ... than to curse the darkness.
Remah is offline   Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off

All times are GMT +1. The time now is 08:52 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2020, vBulletin Solutions, Inc.