File Backup Terminology: What do terms like “Differential,” & “Incremental,” mean, and how will they help me?

Introduction

Over the years various backup technologies have been developed in an attempt to minimize the amount of space required to store backup files, and to reduce the bandwidth required to transfer those files to remote locations.  When faced with the different backup methods that many programs offer, it is easy to become confused, since the terminology used is often not very clear, and it is hard to know the benefits or drawbacks of any one technology.  This article is meant to be a simple guide to help cut down on the frustration that many experience when they don’t know what certain terms mean, and how different options are best used.  

Note: This is not, by far, an exhaustive glossary of backup terms.  If you have questions about any terms that are not covered below, please feel welcome to ask in the comment section, and we will attempt to answer them for you.

Index

Common Backup Methods


Other Backup Methods and Techniques

Discussion

Full Backups

This is just what it sounds like.  This is a complete backup of all the data that a user selects when configuring a backup job. The copied files are usually placed into a single file archive and compressed to help save space.  Every time another full backup is made, all the files in the source are once again copied an archive. The problem is that often there are only a few new or changed files, and continuously making full backups will end up copying a lot of extra files that don’t really need to be backed up again.   This ends up using a lot of extra storage and wastes time.  You can of course delete older backups to free up space, but the time is still lost.  The extra wear on hard disks or the amount of bandwidth that is used to make frequent full backups must be considered too.

It is a much better idea to make a full backup once in a while, and then figure out a way to only copy the new or changed files on a more frequent basis.  Several different methods, described below, have been created to implement this very thing.

Benefits and Disadvantages of Full Backups

  • Faster restore of all files -- When a full restore is necessary full backups are quick because you are only dealing with one archive file.
  • Full backups are large and time consuming to make -- They are not well suited for regular backups such as those performed hourly or daily.

 

Differential Backups

After creating a full backup archive this backup method helps to reduce the size of subsequent backups by doing a “differential” comparison of the original files and the last full backup. All new and modified files are copied to a archive along side the full backup.  The important thing to understand is that differential backups are cumulative.  Each differential backup backs up everything that is different since the last full backup even if those files are already included in a previous differential.  Since Differentials back up only new or changed files, they are a faster backup method than creating a full backup each time.  Differential backups are well suited for daily or less frequent backup strategies.

Differential Backups

Benefits and Disadvantages of Differential Backups

  • Faster to restore that some other methods -- To do a full restore of all backup files, you only need the full backup and the last diff backup.
  • Differential Backups are more demanding on storage than some of the other backup methods, because of data redundancy.
  • Each subsequent differential grows significantly until it becomes necessary to create a new full backup.  Then the process starts over.

 

Incremental Backups

This backup method works similarly to differential backups, but with one important difference that deals with the high level of data redundancy in differentials. Each incremental contains only the files that were created or modified since the last full backup or last incremental.  Incrementals, while not containing as much redundant data as differentials, are still cumulative since successive backups will still contain any files that were already backed but have been modified in some way.  Incremental backups are a good solution for more frequent backups such as those performed on an hourly basis.

Incremental Backups

Benefits and Disadvantages of Incremental Backups

  • Incremental backups can be completed more quickly that differential backups because there is less redundant data being copied.
  • Incremental backups are smaller than differential backups.
  • The number of successive incrementals that can be made between full backups, while still remaining manageable, is much greater than with differentials.
  • Incremental backups may take considerably longer to do complete restores than differential backups because all the individual archives must be merged together one by one with the full backup.
  • It should be noted that a restore from an incremental backup may fail if one of the sequential backups were to be lost or damaged.   Although in all the backups up to the damaged  one should be recoverable.

 

Delta or Block-Level Backups

The term “delta” is often used rather flexibly in reference to different backup technologies, but when paired with other terms as in “Delta Backup,” Delta Block Backup,” and “Delta-Style Backup”  they generally refer to the same basic backup method.   Deltas are best described as  block-level technology, where as incrementals and differentials are file-level technologies.  It is important to note that delta block techniques are only applied to modified files, not new files. New files are of course just backed up in a normal fashion.

File-level backups will backup a changed file in its entirety, even if it has only changed slightly.  While this may not be much of a problem for small text documents, is can quickly become a problem with very large files like databases. Take for example the email clients like Outlook, which save all received email and attachments in single file databases.  Even if only one email has been received, the entire database file has changed, and is backup again.  Since these databases can easily grow to be several hundreds of megabytes in size you once again end up with a lot of data redundancy.

Delta backups deal with this problem by backing up only the parts files which have changed instead of the whole file. Each changed file is broken down in fixed size blocks and those blocks are compared with the original file. (The size of block that is handled is dependent on the particular program or perhaps on a user chosen size.  Block sizes generally range between 1 and 32 kilobytes in size.)  Only those blocks that contain differences are extracted and backed up.  Deltas can be confusing because they can be applied in a couple of different ways.  There are differential deltas, and incremental deltas.  These work on the same principle as the differential and incremental file backups explained above, but at a much more granular level. Similarly each type of delta would inherit the same type of advantages and disadvantages.

Delta Block Backup

Deltas are especially advantageous for use in technologies where files are backed up immediately after files are created or modified. This is known as real-time backup or continuous data protection.  Deltas are also very beneficial when used to backup files over networks with limited bandwidth or to remote servers such as online storage.

Benefits and Disadvantages of Delta Style Backups

  • Delta Backups are extremely fast because of the small amount of data being transferred.
  • Deltas produce much less redundancy, and backups are fractionally smaller than those produced by incremental or differential backups. This dramatically reduces the demands on storage and bandwidth.
  • Deltas of modified files do not produce whole files in the backup, and thus restores absolutely depend on the program that created them to do the restoration.
  • Deltas are slower to restore because the individual files must be reconstructed from their various parts.

 

Binary Patch Backups (FastBit)

Binary patch technology was originally developed as a way for software developers to easily update their programs on customers over the Internet by sending “patches” that would replace the parts of files that needed modification.  Recently it has started to be adapted into backup technologies as well. The most relevant example is a backup technology called FastBittm which is employed by number of online storage vendors.

Binary Patch Backups work very similarly to Deltas, the primary difference being they are even more granular.  Deltas work on a block-level, while binary patches work on the, well, binary level.  Because Deltas backup only the modified parts of files in fixed size blocks, part of that block may contain some unchanged data.  Binary patches avoid this by only copying the actual bytes of the binary code that have changed.

Binary Patch Backup

Benefits and Disadvantages of Binary Patch Backups

Note: Do the very limited application of binary patching technology in actual backup software,  as well as very sparse information on the subject, the author is very uncertain about the benefits and/or limitations that may be inherent to the technique.

  • Virtually eliminates all data redundancy, and produces the smallest backups possible with current technologies.
  • It is even less bandwidth intensive than deltas.
  • The production of the actual patch may be more demanding on system resources and more time consuming than deltas, although the loss may be regained in bandwidth and transfer costs.
  • No information about how file reconstruction is handled and how efficient it is.

 

Mirror Backups

Most backup programs will list mirror backups as an alternative to full, differential, or incremental backups, etc.  Some programs use an alternate term for mirrors, such as “simple copy.”   Mirror backups are basically the simplest type of backup.  There are no real backup technologies being employed when making a mirror style backup, only copy technology.  If you copy and paste a folder from one drive to another you have created a mirror backup of that folder.  The mirrored files generally exist in the same state they did in the source, not compressed into archives like with a full backup.  (Although some programs support compressing each file individually and adding encryption)

When to Use Mirrored Backups
Mirror style backups without compressions are good to use when you are backing up a lot of files with compression already applied them.  For example, music files in mp3 or wma format, images in jpg or png format, videos in dvix, mov, or flv format, and most program install or setup files are already compressed.  If you include these files in a normal backup that applies compression you will often notice it will be very slow, and you will gain very little extra compression by doing so.  It is best to set up separate backup jobs for compressed files and non compressed files.  If your backup program supports include and exclude filters they can be used to either automatically select or deselect the compressed files respectively.

Benefits and Disadvantages of Mirror Backups

  • Mirror backups are much faster when working with compressed files.
  • Because mirrored files are not placed in single archive files there is less concern about corruption.
  • Since mirror backups generally don’t use compression they can require large amounts of storage space, unless other techniques such as hard linking are also employed

 

Synthetic Full Backups

Synthetic Full Backup is a term you will see from time to time and it should be understood that it is not  a backup method like those above, but rather a technology that may be applied to one of the above methods to make full restores more efficient and require less down time.

Synthetics are generally only applied in server - client type backup systems.  A client computer may perform a backup by any method, incremental, delta, etc. then transfer that backup to a server.  At some point the server then combines several of the individual backup archives to form a synthetic full backup.  Because of this, after the initial full backup, the client machine only needs to perform backups of new or modified files, another full backup will never be necessary.

The benefits of this approach are twofold.  First, the backup speed of technologies like differentials won’t degrade over time because of the growing size of cumulative archives since a synthetic will be made on a regular basis.  Secondly, when a full restore needs to be made on a client machine, no reconstruction of files or file parts needs to be done.  The reconstruction has already been performed by the server allowing the client machine the fastest possible recovery time. 


 

Hard Linked Backups (also Hardlink)

Some backup software has the ability to employ multiple hard links to preserve space when you wish to save multiple full mirror style backups of the same set of files. 

To understand what a hard link is consider how files are stored on a hard drive.  When you save a document file, the physical data can be written any where on the disk.  Then the file system makes a reference  or hard link to that physical data with the file name you specify. With some file systems it is possible to create more than one reference to that physical data. Using multiple hard links it is possible to assign any number of file names in different folders to the same physical data.

When using backup programs that support creating  hard links to make several backups of the same files, the program will build hard links for all the files that have not changed.  For example, if you create two copies of a folder that contains 100MB of data, they normally would end up using 200MB of space. With hard links they would only use 100MB of space.  If you changed one 2MB file before you make the second copy using hard links, the two folders would consume 102MB of space.1  The first folder would contain the original 2MB file while the second would contain the modified one.

It should be mentioned that if you decide you want to delete one of the backups containing hard links, it is not a problem, as all the other hard links will be unaffected.  The physical file on the disk is only deleted when all the hard links to it are removed.  Also hard links can only exist within the same volume. ( e.g. they can not span across different partitions or drives)  On Windows based file systems,  NTFS supports hard links, while FAT does not.


1. Windows Explorer does not report file space as one would expect when using hard links.  If a 100MB file has two hard links both links will be reported as consuming 100MB of space for a total of 200MB used.  However, the space saved by the hard links is reflected in the amount of free space on the drive, only 100MB will have been consumed.

 

 

Share this
4.674605
Average: 4.7 (126 votes)
Your rating: None

Comments

by demboy2000 on 27. July 2013 - 2:43  (109653)

Thank you very much for this article. I used this on my presentation.

Thank you very much!

by Beewax (not verified) on 3. September 2012 - 16:39  (98707)

Very well written and easy to understand. You should write the "Backup for Dummies" book. Thank you!

by SKJ (not verified) on 11. July 2012 - 12:53  (96041)

Thanks heaps! This article is absolute gold!!!

by Roderunner on 22. October 2011 - 12:29  (81917)

Their is one backup method missing, 'System Backup' This is the one & only method I use, using [commercial software name removed] 'One Click' method. A backup can be done in under 8 minutes, restore is even faster. Backups are made to a folder named as the date of the backup, including a reminder of how my pc was when made.
I don't need to back up anything else as anything I want to keep is copied to at least 2 external hdd's.
Doing a 'System Backup' before experimenting with new programs is always done, then if it has to be removed, a restore removes it totally whereas a normal uninstall even with Revo Uninstaller does not.

by Bob Stone (not verified) on 23. October 2011 - 4:11  (81967)

I would appreciate knowing what system you use for your back up.
Thanks
Bob Stone

by Ritho on 23. October 2011 - 4:23  (81968)

I personally use Linux as my main OS. Since Linux has no problem working with NTFS partitions I can back up my Windows files from within Linux as well. I use a commercial online service to backup to the cloud, and "Back in Time" to backup to an external hard disk. http://backintime.le-web.org/

by Ritho on 22. October 2011 - 14:33  (81929)

"System Backup" is a very general term, and does not refer to a specific type of backup technology. It usually implies any number of technologies that backup the Operating System or System Partition. In your case, what you seem to be describing, is some sort of disk imaging. Disk imaging is a type of backup technology, and is discussed in another article here. http://www.techsupportalert.com/content/types-backup-software-what-s-dif...

by Urbane.Tiger on 22. October 2011 - 11:00  (81908)

_Hard Linked Backups (also Hardlink)_ is not a backup technique as such. What you've documented are the issues to be considered when files with multiple directory entries (hardlinks) are backed up.

Whilst you hinted at it, file hardlinks on an NTFS drive ordinarily cannot be backed up to FAT or optical drives - some backup programs will do it to some extent, but they cost.

I would have thought you should also discuss the issues of backing up Junctions and SymLinks.

by Ritho on 22. October 2011 - 15:34  (81939)

Thank you for commenting. My section on hardlinks was not meant to define them as a backup method, rather it discusses only how they can be employed to eliminate redundancies between a number of stored archives on the same volume.

Junctions, Symbolic Links, and Soft Links. Are basically synonyms with the following exception that on NTFS volumes, Junctions are used to differentiate between Soft Links that link directories, and Soft Links that link files. All of these only contain information as to the destination of the target directory and/or file, (you could think of it as a "file system shortcut") which is quite different from a hardlink.

I am quite willing to stand corrected, but I do not know of any backup software that employ Soft Links in any special manner. In regards to backing up the file to which a soft link points, I do not think it is different than backing up any other file.

by akakingess on 17. April 2011 - 16:56  (70394)

Thank you, thank you, thank you, finally a clear explanation of the backup utilities terminologies and/or protocols. Sometimes I think some software companies just make stuff/terms up to make themselves seem smarter or make it sound like their software is the only one that can do 'whatever'. At least so far I haven't seen the ol' "Our software will simply read your mind and back up everything just the way you want it" line yet, but I feel it is coming soon. So, thank you again very much for clarifying the terms for me and keep up the good work. Now I am off to buy a Gizmo's cap (they are very cool looking, much less the fact that they help support good guys like y'all). Please keep writing and I will be coming back.

Earl
a.k.a. KingESS

by Doron (not verified) on 29. March 2011 - 22:00  (68803)

What windows software do you recommend for hardlink backups?

by Ritho on 30. March 2011 - 5:30  (68812)

I am not aware of many so other's suggestions are welcome. Here is one I reviewed on the site a while back. http://www.techsupportalert.com/content/datasafe-backup-average-backup-s...

If you decide to use it do yourself a favor, and use a file integrity verification utility such as the one suggested in the article.

Ritho
Editor

by Roberta (not verified) on 28. March 2011 - 23:08  (68718)

Excellent article!!

Thank You Very Much,
Roberta

by Denis Gauthier (not verified) on 28. March 2011 - 21:31  (68712)

Excellent explanations.

I use for years GFI Backup (formerly Titan Backup). When editing tasks, it's important to understand differential and incremental concepts.

by Ian from South Florida (not verified) on 28. March 2011 - 20:20  (68710)

Sigh! I have been waiting for years for Microsoft to dived the hard disk into two partitions:1 the operating system and 2 the applications and data.

With this simple change the operating system could be backed up and / or restored without destroying user data.

Yes it would mean two backups but the benefits and convenience of just doing a system restore out weigh the effort.

by DonsEars (not verified) on 28. March 2011 - 17:18  (68702)

I have never been a big fan of mirrors or images. They tend to restore you right back to your problem. (Assuming you backup often. Images are good for hardware failure, but will only take you back to the last time you did an image. Which was probably when you got that new PC.) And when you accidentally delete a file, it is then purposely deleted on the mirror.

I have been looking for "versioning" software similar to Dropbox.com or Apple Time Machine for my documents folders. I want the backup software to do a full backup and when I add files only the new files are copied. When I change or delete a file, the software moves the original backup file somewhere else in case it needs to be recovered.

One important feature for me since getting hosed by proprietary software is that the backup is in the same format as the master. Simple file browsing and copying will restore the files. Many years ago, PC Tools was notorious for obsoleting your backups with every new release.

I guess the bottom line is to have a disk image for hardware failure and file backups that can be rolled back a few days for human error and virus protection. Oh, yes. And important file copies stored at a physically different location.

by sam Craig (not verified) on 24. December 2011 - 1:49  (85844)

I have been looking for "versioning" software similar to Dropbox.com or Apple Time Machine for my documents folders. I want the backup software to do a full backup and when I add files only the new files are copied. When I change or delete a file, the software moves the original backup file somewhere else in case it needs to be recovered.

Does anyone know any software which do the above functionality.

by alex6500 (not verified) on 28. March 2011 - 18:23  (68705)

On my ext hard drive i first did an image backup.I then partitioned the ext.hard drive into 3 partitions and one of the new partitions uses an incremental program.The 3rd partition is for anything else. The only disadvantage on this setup is if i want to do an image backup the ext drive is converted back to one partition and the incremental backup is lost.I found this out by doing the image backup after i did
an incremental backup.I am no pro but this works for me.

by Zepe (not verified) on 28. March 2011 - 15:43  (68693)

I keep two external HDs, one is my primary backup and the other is my secondary. The contents of these two drives are the same. Each has an image of the C drive (which only contains the OSs and applications) and a copy of my files . My files are kept on a different partition on the computer and are backed up regularly (depending on how many changes and/or additions I've made) with Karen's Replicator. I use Acronis for the hard drive images which I replace monthly.

by Dwaine (not verified) on 28. March 2011 - 15:07  (68691)

The best backup utility I have found is "Robocopy.exe" built into Win7 & Vista and available for XP in the "Windows Resource Kit Server 2003" and NT4 in the "NT4 Recourse Kit".

The syntax is: ROBOCOPY source destination /MIR

First time run it mirrors, 2nd-on time run it only adds/deletes files that have changed. It runs from the command line but don't let scare you go ahead and learn it.

I have been using the program for years and it is fast and well proven.

by alex6500 (not verified) on 28. March 2011 - 18:27  (68706)

I like your setup.

by Snert (not verified) on 28. March 2011 - 13:22  (68688)

This artice was clear and concise, the way I like 'em.
Myself, I do a full backup to an external HD once a month, JIC. I save the new backup and delete the old one.
I don't need to do anything else (crosses fingers).

by ouman77 (not verified) on 28. March 2011 - 13:14  (68687)

I read the article and am still bemused/confused (which is not hard at my age - I can even successfully hide my own Easter eggs!)
What I need is a program that will back up my SYSTEM disk (with the system, registry &c. and all my programs), to an external disk, so that - if my system crashes - I can roll back to a fully functional PC within the matter of ½ to 1 hour. (not the weeks it takes to try to get everything "back to normal".)
What sort of backup/program would I need to be looking at?
I just intend to do the backup fortnightly, deleting any previous backups, to "keep it lean".

by Nic (not verified) on 7. April 2011 - 4:41  (69601)

LOL!!! Still giggling after 10mins about being able to hide your own Easter eggs.. April must be a very exciting month for you. Thanks for posing this scenario, it is exactly what i want to achieve also - doing snap mirror images is fine for total restore, but useless if you want to recover a single file.

by Rob (not verified) on 28. March 2011 - 13:36  (68689)

See my very recent post about 'KISS' and the free Seagate DiscWizard. That will do exactly what you requested.

PS for others,
The DiscWizard program is by Acronis, that has been made simple and robust by Seagate.

by Rob (not verified) on 28. March 2011 - 12:40  (68684)

- - - - Or you could use 'KISS' - - - -
Get an external Dock (get two, as they are cheap).
Buy two Seagate Sata 3.5" drives, and shove them vertically into the Docks. Get Docks that have both USB and eSata connectors. You will know if you are getting the correct Docks, as the 3.5" drive will be sitting vertically, with three quarters of it exposed to the nice cool air.
Get the free Seagate DiscWizard program, and use it to create a bootable CD.
Connect the Dock, and boot into the CD. (Do not have Windows running.)
Create an image of the whole drive.
Verify the image (which is another option in the DiscWizard screen).
The next time that you create an image, do it to the other Dock. You use the Docks alternately for your backups.
If you have vital data that you would slash your wrists if lost, then back that up more frequently to CD, or thumb drive, or another drive.
Regarding that last comment, I am referring to say a manuscript that you are typing, or a project that you are coding.
Don't muck about with incremental, differential, etc backups.

"Now who could argue with that"
(Blazing Saddles, in the church)

by chesscanoe (not verified) on 28. March 2011 - 12:31  (68683)

Really good article and the graphics reinforce the concepts well. As a home user of Win7 Pro x64 SP1, I find that a monthly Win7 provided image backup meets my needs. It's easy to do overnight the Saturday after patch Tuesday, and if I do need to recover in the future, odds favor only two weeks of data lost, which I can live with. Not a geek solution by any means, but it's easy to administer.

by Ouatcheur (not verified) on 24. March 2011 - 17:44  (68504)

How do you call an backup that results in the same thing as a full or mirror backup, but that merely updates directly ONLY whatever files that changed from the original source's original full backup, rather than creating separate data?

Not this:
-Day 1 complete backup of drive D resulting in complete backup #1 on drive E.
-Days 2, 3, 4, etc. changes-only backup of drive D resulting in the same unchanged complete backup #1 on drive E + incremental/or/differential backups #2 #3 #4 #etc. on drive E, constantly growing in numbers (and thus space needed) until a new full is needed.

But this:
-Day 1 complete backup of drive D resulting in complete backup #1 on drive E.
-Days 2, 3, 4, etc. performing a CHANGES-ONLY backup of drive D resulting in only an update of the only backup on drive E so that now backup #1, will always be updated to contain the same files content of drive D, exactly the same as if had done nothing on all previous days, and done the complete backup only the last day.

In short, doing changes-only backup that ends up looking like a complete backup (albeit at the sacrifice of the _previous_ complete backup)Slower than incremental, sure, but as easy to restore as a full, and that doesn't need more space than a full.

For home use, where there is true need for a backup, but often no need for OLD backups, this would be perfect, combining most of the advantages of full (or mirror) with incremental.

How is that kind of backup called?

i.e. I want the speed of incremental backups, but without the hassle of the extra space needed or of the complexity of rebuilding everything afterwards in case of a problem that incremental and differential have. In short, what I really want is a mirror copy of all files, that is then afterwards simply "maintained" up to date, not rebuild from scratch each time!

by Ritho on 24. March 2011 - 18:33  (68506)

If I understand correctly, what you are asking about is just another type of Synthetic Full Backup. (See Above) Sometimes this is simply called a "Synthetic Backup." Another term that has become fairly common is "Virtual Full Backup."

I don't know of any free backup programs that will do it except for Bacula http://www.bacula.org/en/. It is probably a bit much for normal home use, but it has a lot of powerful features and is used by a lot of large companies.

Edit: Apparently BackupPC will also handle making synthetic backups http://www.backuppc.com/

by jayell (not verified) on 5. March 2011 - 3:14  (67452)

Frank D said it for me: thanks for clearing up a lot of confusion for me!

Gizmos Needs You

Gizmo's Freeware is Recruiting

 We are looking for people with skills or interest in the following areas:
 -  Mobile Platform App Reviews for Android and iOS
 -  Windows, Mac and Linux software reviews       Interested? Click here