Attention VMware experts: Seeking quick way to back up a 2TB VM
January 23, 2017 8:19 AM   Subscribe

It currently takes about 3½ days to back up my 2TB virtual file server. I need to find a faster way to do this.

I work for a small organization. My VMware setup consist of four boxes: three white-box ESXi hosts and a vCenter server. All are connected to a gigabit switch. The license is VMware Essentials (regular Essentials, not the “Plus” version). My backup application is Veeam Backup Essentials Standard.

I recently virtualized our 2TB file server in order to facilitate making backups and possibly recovering from a disaster (esp. ransomware). The problem is that backups of the VM take way too long. Backing up to a NAS (which is also connected to a gigabit switch) takes more than three days. Backup speeds top-out at around 10 megabytes per second. Veeam Backup lists the bottleneck as “Source”.

I’d like to implement a low-cost, automated way to make a daily backup of the whole VM, then copy the backup to one of several external hard drives that would be rotated to an off-site location. What’s the best way to do this on a budget? I’m guessing I need to set up some kind of fiber-optic network to connect the various VMware devices and the NAS (or perhaps a full-blown storage server rather than a NAS). But when I start Googling possible solutions, all I can find are configurations that are expensive and are probably gross overkill for what I need.

The current backup system would work great for me, if it completed in a couple of hours instead of a couple of days. So I’m really just looking for a way to increase the speed. Can someone steer me in the right direction? As you might have guessed, I am very much not an expert on VMware, so please don’t assume too much knowledge on my part.
posted by alex1965 to Computers & Internet (14 answers total) 3 users marked this as a favorite
A gigabit network should be able to do about 10 times what you're getting, so if the network is really the bottleneck then something's broken. And even a single drive should be able to read and write faster than that.

I'm unclear--were you backing up your file server at all before virtualizing it? Did it perform better before?
posted by floppyroofing at 8:38 AM on January 23, 2017

Do you get normal gigabit speeds off the fileserver in other use cases (i.e. as a file server and not a backup client)? That would help narrow down whether it's a vNic problem or a vDisk problem.
posted by Kyol at 8:41 AM on January 23, 2017

@floppyroofing: Before virtualizing the file server, I was using Robocopy to back up just the files. An incremental backup would take less than an hour. The backup procedure was working, but restoring from backup was problematic, because first I had to rebuild the whole file server, then configure it the same way as the old server, and then restore all the files to it. I'm looking for a faster way to restore, in the event of some kind of disaster.

@Kyol: I haven't measured it. That's a good idea.
posted by alex1965 at 8:44 AM on January 23, 2017

You could put the file server data partition on the NAS and mount it in esxi with NFS or SMB. We've migrated away to Xen Server in the last year or three, but our legacy esxi servers still have this setup for some shared VM data. You could do the same remote mount inside your VM, but doing it in esxi makes it show up like any other local drive as I recall (and for our situation lets us share the same "drive" among multiple VMs).
posted by mattamatic at 8:47 AM on January 23, 2017

Does Veeam not support change block tracking backups? Turn on change block tracking, complete your 3 day backup once and all subsequent backups should be lightning fast.
posted by Cosine at 8:53 AM on January 23, 2017 [5 favorites]

I'd check on the VMWare instance's network settings, particularly the network adapter as documented here. This sounds like a bad network driver or misconfigured interface to me, although it could be a number of things not specific to VMWare. I've run into the MTU size problem a few times in software over the last few years and have fixed a few of them by bumping the frame size.
posted by mikeh at 9:08 AM on January 23, 2017

Veeam with change block tracking should be able to back this up in minutes to hours rather than days. I have an Exchange server with 2TB of mail databases that backs up every night with CBT on. (Cosine is dead on)

Also with source being slow, what kind of drives are in the ESXi host? How old are the drives? Are they all healthy?
posted by deezil at 9:32 AM on January 23, 2017

Because I didn't know about change block tracking, I googled around a bit; this looks informative
posted by floppyroofing at 10:16 AM on January 23, 2017

Do your VMware hosts use local storage or shared storage? What type of Veeam Proxy are you using? Is your backup mode is defaulting to "nbd"?

Try installing a "Virtual Appliance" backup proxy. Before we got our SAN and I switched to Direct Storage Access, Virtual Appliance mode was way faster than Network mode.

Make sure cbt is enabled.
posted by LoveHam at 10:17 AM on January 23, 2017

Thanks for the tips about change-block tracking. I will look into it.

@deezil: The drives are 4TB Western Digital Reds. They are just a few months old.

@LoveHam: The VMware hosts use local storage. I don't have any shared storage. Not sure what you mean by the type of Veeam Proxy. I think the proxy is just the vCenter server on which the Veeam Backup application is running. I'm also not sure about the backup mode -- I don't see "Backup Mode" listed as a setting anywhere in the backup job definition. A few minutes on Google didn't turn up anything that told me where I can change it on Veeam Backup Essentials. I'm assuming that the backup is going over the network, since there is no other path that the information can take.
posted by alex1965 at 10:50 AM on January 23, 2017

Is a robocopy backup slower than it used to be?
posted by gregr at 11:12 AM on January 23, 2017

@gregr: No, I haven't noticed any change in speed for the robocopy backup.
posted by alex1965 at 11:22 AM on January 23, 2017

Sounds like the Veeam server is your only proxy. That means you're using Network Mode, which is the slowest.

To add a Virtual Appliance/hotadd proxy, create a new Windows VM (they recommend a server OS for this but I used to use Windows 7) on the same VMware host as your file server. Then make that new VM a backup proxy assigning the new VM the hotadd proxy role in the wizard. Veeam is smart enough to use the best proxy available so should chose the new proxy for the next backup.

If you're still unsure on how to add the new proxy, give Veeam support a call. They are very helpful.
posted by LoveHam at 2:08 PM on January 23, 2017

Probably a dumb question-- you mention the switch is Gigabit, but are the PC's NICs also Gigabit?
posted by Static Vagabond at 10:35 AM on January 24, 2017

« Older Need help with rejecting helpful offers of...   |   What are some great political activism tools for a... Newer »
This thread is closed to new comments.