A New Backup Architecture - SnapDiff V3
Ling Zheng
The challenges of NAS backup
With the explosive growth of data, backing up NAS systems is increasingly challenging and complex for enterprises around the world.Historically, NAS data was backed up and restored with the Network Data Management Protocol (NDMP). However, NDMP architecture is inherently inefficient and outdated.
With NDMP, backups are typically performed as periodic full backups with intervening incremental backups. To restore backed up data to a specific point in time, NDMP needs to restore the last full backup before the restore point and then apply data from all the incremental backups between the last full backup and the restore point. As the number of incremental backups increases, the ability to restore becomes increasingly challenging, complex, and time consuming. As a result, NDMP backup operations require regular full backups, which increase the customer’s backup window. These inherent inefficiencies make NDMP difficult to scale to meet the backup needs of large enterprises.
NDMP was originally designed to work with tape-based backup media. Tape systems are labor intensive to manage and difficult to scale. Many enterprises are adopting cloud storage to meet their backup needs. Instead of maintaining dedicated tape subsystems or purpose-built backup appliances, customers are moving to general-purpose public cloud object storage and private cloud infrastructure as the technology of choice for storing their backups. Object storage offers ease of scaling to any capacity and ease of access from any location, as well as the flexibility of a “pay-as-you-grow” model.
A new backup architecture
To help customers solve the NAS backup problem and adopt the latest technology, NetApp is introducing a new SnapDiff v3 backup architecture to efficiently back up NAS data to both on-premises and cloud object storage across the data fabric. SnapDiff v3 architecture is highly efficient and cloud friendly for backing up NAS systems.NetApp® ONTAP® customers have long used NetApp Snapshot™ and SnapMirror® technologies to create efficient Snapshot copies to transfer between ONTAP systems. The new SnapDiff v3 architecture leverages the same unique underlying ONTAP capability to efficiently identify changed data between ONTAP Snapshot copies and then transfer it to a backup target.
Two key enabling technologies make up the SnapDiff v3 architecture:
- SnapDiff, a differencing engine that generates a list of changed files between two Snapshot copies.
- SnapMirror Cloud, an extension of SnapMirror that transfers Snapshot copies from an ONTAP system to an object storage target.
This architecture offers the following benefits:
- Forever incremental backup. After the initial full backup, no more full backups are required.
- Block-level incremental. If one block of a file is changed, only that block is backed up.
- Storage efficiency preservation. Deduplication and compression in ONTAP are preserved so that backups are transferred and stored most efficiently.
- Granular file recovery. Files can be recovered individually from object storage.
- Online backup data access. Backup data in the object storage can be accessed directly for analytics and other use cases.
SnapDiff v3 backup architecture.[/caption]
Backup workflow
The following is a typical backup workflow using SnapDiff v3 architecture:- Identify and configure an object storage (for example, a bucket created in cloud storage or NetApp StorageGRID®) where backup data will be sent.
- Create a SnapMirror policy for the object storage.
- Select an ONTAP volume to be protected, then create a SnapMirror relationship between the ONTAP volume and the object storage target by using the SnapMirror policy.
- Start the data transfer—an initial full baseline backup followed by incremental transfers.
- Run SnapDiff along with each transfer to generate a list of files that have changed since the previous backup and update the backup catalog.
Restore workflow
Backup data in object storage can be restored back to ONTAP or accessed directly in a number of different ways. The following is a workflow to restore individual files using SnapDiff v3 architecture:- Select a list of files from the previous SnapDiff output to be restored.
- Start the data transfer from the object storage to an ONTAP volume to restore those individual files.
Summary
SnapDiff v3 architecture addresses the key challenges that companies face with NDMP backups. With SnapDiff v3, ONTAP users can improve backup operations and reduce backup windows by using a combination of SnapDiff and SnapMirror Cloud to efficiently identify and transfer changed data to object storage on the premises and in the cloud.NetApp is working with selected partners to develop solutions based on the new backup architecture. For more information about the SnapDiff v3 API program and to participate in the SnapDiff v3 API Technology Preview, please contact your NetApp representative.
Updated on 1/26/2021
SnapDiff™ v3 have been officially released in ONTAP 9.8. See the list below for the partners working with NetApp on the integration. Click to see the blog post by NetApp’s partner ProLion on the SnapDiff v3 and SnapMirror Cloud integration.
- Click to see the product page from Cleondris on the SnapDiff v3 and SnapMirror Cloud integration.
- Click to see the blog post by Druva on the SnapDiff v3 and SnapMirror Cloud Integration.
