Cloud Backup and DR – The Problems with Seeding Data

ByNovember 2, 2016

initializing - initial syncWhat is data seeding?

Many companies are experiencing the benefits of cloud Backup and Disaster Recovery solutions with the use of Veeam Backup and Replication, Zerto and Double-Take. However, each of these solutions require the first initial backup or replica of servers – and this can be a huge speed bump for many customers. Often, customers will request to seed the data in an effort to speed up the initial sync and I’ve had a lot of experience helping customers through this process at iland. Usually, this can be accomplished by creating backups or transferring VM files to an encrypted disk provided by iland. This disk will be sent to iland and the VM files, backups or replica data will be transferred to iland servers and used as a seed for the initial backup/replica.

In theory, this seems like a quick workaround for customers with limited bandwidth and/or customers with a large amount of data to replicate. However, many times this seeding process has created more headaches and delays to replication than desired. iland still uses this method in certain use cases but seeding has become a last resort for new Backup and Replication customers.

Why would I want to try and manually seed my replica/backup?

Regardless of which solution you use, the fact is you have to replicate or backup your servers for a first time. This process generally requires a significant amount of time which is dependent on the amount of bandwidth you have available and the amount of data you need to replicate.

A good benchmark to use is if you have 100Mbps of bandwidth available, you can typically replicate or transfer about 1TB per day.

You can use online tools to estimate the amount of time required for a backup or replica, such as this Cloud Calculator.

Let’s work through an example – say you have 10 servers that total up to 5TB of used storage and have a 100Mbps of bandwidth. Technically, it’s possible to replicate all of these servers within a week. However, many customers must split their available bandwidth and only reserve a smaller amount for their production environment. DR solutions can use a large amount of bandwidth, especially on their initial syncs, and this may cause issues for your end users or employees during work hours. So, for your 5TB of data to replicate, you may only be able to allocate 50Mbps or 25Mbps of your bandwidth to replication. This will cause the initial sync to run for 10 or 20 days. If you are using Zerto or Double-Take, you can expect this to take even longer as these solutions use real time replication. So, as your servers are replicating their data for the first time, new data created or changes made to this server get added to the amount of data to be replicated. For some customers, it’s even possible that the progress moves backwards for replication as new data is being created faster than it can be replicated.

For Veeam customers, Veeam creates a snapshot and only replicates that image of the server but new changes are not added to the current job. This means that the servers being replicated will have a snapshot on them for several days and may cause a performance hit once it is removed. If this Veeam Replica/Backup takes 5 days, that means the second run will have 5 days’ worth of changes to replicate, which can be a significant amount. This time can be longer/shorter depending on the compression and deduplication utilized during a replica or backup. Regardless, the initial replica can be very time consuming and some customers may not have the bandwidth available or time to work through it.

What are the problems with Data Seeding?

Many customers who deal with the issues described above see data seeding as the only possible solution to complete their initial syncs. This is often due to time constraints. They need to have a DR environment or backup ready to use before a specific dead line. Other issues might be that they are not able to complete the initial sync with their bandwidth vs. the amount of space they need to replicate or backup.

For customers trying to complete the initial sync faster with data seeding, the seeding process can often take a longer time. How is this possible? The first step for the seeding process is for iland to ship out an encrypted storage device, usually a disk, QNAP or other SAN device. Once the disk arrives at your office or datacenter, this data will need to be transferred onto an external device. There are many methods that can be used for this, whether it’s a VM backup with Veeam or another application, OVA export, or just copying VMDK and VMX files from your VMware infrastructure onto the disks. How you transfer these files to the disk will determine the amount of time needed.

For our 10 VM, 5TB of used storage example above, a transfer with a USB 2.0 device would take about 23 hours to transfer all of the data. Let’s say that all data gets transferred on the disk October 17th, the next step is to ship this drive to an iland datacenter. With a time constraint, you may wish to use overnight shipping, which can incur considerable additional costs. Once it arrives to an iland facility, we then have to mount the disk at our datacenter and transfer the data. Ideally, the data will be at the target site and ready to seed October 19th but this time could be longer depending on the time to ship the device, when the disk is able to be mounted, and the transfer from the disk to the target repository or data stores. The last step will be to set up the replica job to map the data.

Once all of that is set, we are ready to begin the first replica/backup with the seeded data. However, the seeded data is now at least 2 or 3 days old at this point. Veeam, Zerto and Double-Take all have a “Mirroring” type process that will be performed during a seed where the software will compare the Source Server with the Target Server. This requires checking each block on the target and source, finding the differences and eventually replicating any changes.

So, if this is a Zerto replica, you will see a Delta Sync process starts, and once this starts you are not able to failover until it completes. Zerto is comparing both sides, matching the differences but also adding in changed data just as it does with the initial sync. Again, this process is dependent on the , thankamount of data seeded, the amount of changes and available bandwidth. So for my 5TBs, we can say about 500GB worth of data has changed (about 10% of my used data). If I can only allow 25% of my bandwidth to be used in replication, then I can expect the Delta Sync to take at least 2 days of replication for that 500GB after Zerto has compared both sides. Again, I also have to deal with current changes being added to this as well. Essentially, after paying for shipping and datacenter costs, waiting for data to transfer and finally seeding my replicas, I may be in the same situation as with the initial sync.

How do I complete an Initial Sync without Seeding?

If I don’t use seeding, then how do I complete the initial sync? Bandwidth, Optimization and Patience. The easiest solution, but sometimes most difficult,is to upgrade your bandwidth. Obviously, the bigger pipe you have the faster you can replicate and continue replication when adding more and more servers. However, upgrading may not be possible for some customers. In that case, you can work with iland to try and optimize the solution during your initial syncs. We can work to configure the backup or replication job to allocate the bandwidth to specific servers and stagger the initial syncs. For instance, instead of trying to replicate all 10 Servers and 5TBs of data in my example, we can focus on 2 or 3 VMs at a time to complete a sync and progress from there. It may also be best to wait until a weekend when you can remove any throttles and use all of the available bandwidth without affecting the production end user’s performance. Once the replication or backups have been optimized and scheduled as best as possible, the last option is to give the job time and monitor.

For Veeam customers, you will want to ensure that your local job and offsite jobs to iland do not overlap. If you are replicating a server to iland, a local backup on that same server could cause snapshot issues or break the replica job. Similarly, when performing a backup or backup copy job to iland, a local backup may lock the VBK files of the server and cause the job to iland to break. Refraining from local backups or replicas while sending the data will prevent Veeam from locking backup files that might cause a job failure.

Support and advice is key for successful setup

However, iland typically suggests first trying to seed over the wire if at all possible to avoid further complications and delays. The initial sync process can be a time and resource consuming process but it is an essential step for your Disaster Recovery and/or Backup Solution. When beginning the replication and backups with iland, our support team is always available to help and make recommendations to complete this process as quickly and easy as possible. Even with the possible issues involved in data seeding, this is still always an option.
Mike Mosley

Mike Mosley

Mike Mosely is a cloud engineer at iland and has worked at the company for over 3 years. He holds a number of VMware certifications including VCP5 as well as the Veeam VMCE certification. Mike works closely with customers to build cloud solutions that fit their requirements.