Quick Way to Migrate VMs Between Standalone ESXi Hosts

Introduction

Since vSphere 5.1, VMware offers an easy migration path for VMs running on hosts managed by a vCenter. Using Enhanced vMotion available in Web Client, VMs can be migrated between hosts, even if they don’t have shared datastores. In vSphere 6.0 cross vCenter vMotion(xVC-vMotion) was introduced, which no longer requires you to even have old and new hosts be managed by the same vCenter.

But what if you don’t have a vCenter and you need to move VMs between standalone ESXi hosts? There are many tools that can do that. You can use V2V conversion in VMware Converter or replication feature of the free version of Veeam Backup and Replication. But probably the easiest tool to use is OVF Tool.

Tool Overview

OVF Tool has been around since Open Virtualization Format (OVF) was originally published in 2008. It’s constantly being updated and the latest version 4.2.0 supports vSphere up to version 6.5. The only downside of the tool is it can export only shut down VMs. It’s may cause problems for big VMs that take long time to export, but for small VMs the tool is priceless.

Installation

OVF Tool is a CLI tool that is distributed as an MSI installer and can be downloaded from VMware web site. One important thing to remember is that when you’re migrating VMs, OVF Tool is in the data path. So make sure you install the tool as close to the workload as possible, to guarantee the best throughput possible.

Usage Examples

After the tool is installed, open Windows command line and change into the tool installation directory. Below are three examples of the most common use cases: export, import and migration.

Exporting VM as an OVF image:

> ovftool “vi://username:password@source_host/vm_name” “vm_name.ovf”

Importing VM from an OVF image:

> ovftool -ds=”destination_datastore” “vm_name.ovf” “vi://username:password@destination_host”

Migrating VM between ESXi hosts:

> ovftool -ds=”destination_datastore” “vi://username:password@source_host/vm_name” “vi://username:password@destination_host”

When you are migrating, machine the tool is running on is still used as a proxy between two hosts, the only difference is you are not saving the OVF image to disk and don’t need disk space available on the proxy.

This is what it looks like in vSphere and HTML5 clients’ task lists:

Observations

When planning migrations using OVF Tool, throughput is an important consideration, because migration requires downtime.

OVF Tool is quite efficient in how it does export/import. Even for thick provisioned disks it reads only the consumed portion of the .vmdk. On top of that, generated OVF package is compressed.

Due to compression, OVF Tool is typically bound by the speed of ESXi host’s CPU. In the screenshot below you can see how export process takes 1 out of 2 CPU cores (compression is singe-threaded).

While testing on a 2 core Intel i5, I was getting 25MB/s read rate from disk and an average export throughput of 15MB/s, which is roughly equal to 1.6:1 compression ratio.

For a VM with a 100GB disk, that has 20GB of space consumed, this will take 20*1024/25 = 819 seconds or about 14 minutes, which is not bad if you ask me. On a Xeon CPU I expect throughput to be even higher.

Caveats

There are a few issues that you can potentially run into that are well-known, but I think are still worth mentioning here.

Special characters in URIs (string starting with vi://) must be escaped. Use % followed by the character HEX code. You can find character HEX codes here: http://www.techdictionary.com/ascii.html.

For example use “vi://root:P%40ssword@10.0.1.10”, instead of “vi://root:P@ssword@10.0.1.10” or you can get confusing errors similar to this:

Error: Could not lookup host: root

Disconnect ISO images from VMs before migrating them or you will get the following error:

Error: A general system error occurred: vim.fault.FileNotFound

Conclusion

OVF Tool requires downtime when exporting, importing or migrating VMs, which can be a deal-breaker for large scale migrations. When downtime is not a concern or for VMs that are small enough for the outage to be minimal, from now on OVF Tool will be my migration tool of choice.

Advertisement

Jumbo Frames justified?

When it comes to VMware on NetApp, boosting  performance by implementing Jumbo Frames is always taken into consideration. However, it’s not clear if it really has any significant impact on latency and throughput.

Officially VMware doesn’t support Jumbo Frames for NAS and iSCSI. It means that using Jumbo Frames to transfer storage traffic from VMkernel interface to your storage system is the solution which is not tested by VMware, however, it actually works. To use Jumbo Frames you need to activate them throughout the whole communication path: OS, virtual NIC (change to Enchanced vmxnet from E1000), Virtual Switch and VMkernel, physical ethernet switch and storage. It’s a lot of work to do and it’s disruptive at some points, which is not a good idea for production infrastructure. So I decided to take a look at benchmarks, before deciding to spend a great amount of time and effort on it.

VMware and NetApp has a TR-3808-0110 technical report which is called “VMware vSphere and ESX 3.5 Multiprotocol Performance Comparison Using FC, iSCSI, and NFS”. Section 2.2 clearly states that:

• Using NFS with jumbo frames enabled using both Gigabit and 10GbE generated overall performance that was comparable to that observed using NFS without jumbo frames and required approximately 6% to 20% fewer ESX CPU resources compared to using NFS without jumbo frames, depending on the test configuration.
• Using iSCSI with jumbo frames enabled using both Gigabit and 10GbE generated overall performance that was comparable to slightly lower than that observed using iSCSI without jumbo and required approximately 12% to 20% fewer ESX CPU resources compared to using iSCSI without jumbo frames depending on the test configuration.

Another important statement here is:

• Due to the smaller request sizes used in the workloads, it was not expected that enabling jumbo frames would improve overall performance.

I believe that 4K and 8K packet sizes are fair in case of virtual infrastructure. Maybe if you move large amounts of data through your virtual machines it will make sense for you, but I feel like it’s not reasonable to implement Jumbo Frames for virual infrastructure in general.

The another report finding is that Jumbo Frames decrease CPU load, but if you use TOE NICs, then no sense once again.

VMware supports jumbo frames with the following NICs: Intel (82546, 82571), Broadcom (5708, 5706, 5709), Netxen (NXB-10GXxR, NXB-10GCX4), and Neterion (Xframe, Xframe II, Xframe E). We use Broadcom NetXtreme II BCM5708 and Intel 82571EB, so Jumbo Frames implementation is not going to be a problem. Maybe I’ll try to test it by myself when I’ll have some free time.

Links I found useful:

• ESX Server, IP Storage, and Jumbo Frames
• VMware KB: iSCSI and Jumbo Frames configuration on ESX/ESXi
• NetApp and VMware Virtual Infrastructure 3 Storage Best Practices
• VMware vSphere and ESX 3.5 Multiprotocol Performance Comparison Using FC, iSCSI, and NFS

Overclocking Q6600 on P5B Deluxe

There are hellova articles in the Internet on OC in general and particularly on overclocking CPUs on ASUS P5B Deluxe and other P965 chipset motherboards.  So here I’ll just post my results and BIOS configuration, along with some of my findings.

I was able to overclock FSB from 266 to 400 with x8 CPU multiplier. Which effectively means 3200MHz frequency as opposed to 2400 stock.

BIOS settings

JumperFree Configuration

CPU Frequency: 400
DRAM Frequency: DDR2-800MHz
PCI Express Frequency: 101
PCI Clock Synchronization Mode: 33.33MHz
Spread Spectrum: Disabled
Memory Voltage: 1.9V
CPU VCore Voltage: 1.525V
NB VCore: 1.25V
SB VCore (SATA, PCIE): 1.5V
ICH Chipset Voltage: 1.057V

CPU Configuration

CPU Ratio Setting: 8
C1E Support: Disabled
Max CPUID Value Limit: Disabled
Vanderpool Technology: Enabled
CPU TM Function: Enabled
Execute Disable Bit: Enabled

North Bridge Configuration

Basic Timings: 5-5-5-15-5
Additional Timings: 42-10-10-10-25
Static Read Control: Disabled

The most important thing to realize about OC on P5B Deluxe is the necessity of manual memory timings set up. Initially I wasted a lot of time trying to OC over 350 FSB with no luck. After changing timings to mentioned above I easily OCed to 400. I can’t explain why you need to set them like this. I just found them in the Internet and it works for me.

What was also new for me is that CPU with x6 multiplier and 400 FSB won’t work with the same voltage as x9 on 266. It’s the same frequency, however, CPU always init on x9 multiplier and only after power up system changes it to configured in BIOS. It means that if you want to lower CPU voltage by changing multiplier, then don’t expect voltage to decrease to initial values.

Another interesting fact is that with C1E Support setting enabled you will get less performance and less marks in CPU dependent benchmarks like 3DMark. C1E Support can lower CPU frequency when CPU idles. But it seems that it also reduces its performance under load.

I also left CPU TM Function (throttling) enabled for safety purposes.

For those who want to increase FPS in games I want to say that CPU and memory OC won’t give you any significant performance boost in games. For example in Unigine benchmark for 266 FSB, 2.4GHz CPU I get 37.9 average FPS. And for 400 FSB, 3.2GHz I get 38. I agree that for some games it will make difference but it’s not worth it. Don’t torture your system.

Results

266 FSB, 2.4GHz CPU:

Max temperature in linpack (OCCT) 54C (measured by Core Temp)
SiSoft Sandra Memory Throughput 5Gb/s
3DMark’06 12211
Unigine Heaven 37,9 avg. FPS
1:10 hour HD video encoding 3:47 hours

400 FSB, 3.2GHz CPU:

Max temperature in linpack (OCCT) 87C (measured by Core Temp)
SiSoft Sandra Memory Throughput 6.45Gb/s
3DMark’06 14945
Unigine Heaven 38 avg. FPS
1:10 hour HD video encoding 2:59 hours

As you can see I have extreme temperatures in linpack, even though I have Thermaltake Big Typhoon VX cooler and efficient Arctic Cooling MX-2 thermal grease. However, you should understand that system never gets to these temperatures under standard system loads like gaming, video encoding, etc. Usually I get no more that 65-70 which is not that high.

Bare in mind that P5B Deluxe undervolt your CPU by 0.5V. It means when you set VCore to 1.525 it’s actually 1.475. Also when I set VCore higher than 5.125 (1.465 effective) motherboard automatically changes VCore voltage to Auto. In fact, it will work on set up voltage until you enter BIOS and save changes for the second time. In other words if you set voltage higher than 5.125 then you will need to set it again after you enter BIOS and change anything for the next time.

The main reason for me to OC my system was video encoding. Firstly I changed old 2 core E6300 to 4 core Q6600 and then OCed it. I used 1:10 hour HD video for testing purposes. Time has changed from 8:08 on stock E6400 to 3:47 on stock Q6600 and then to 2:59 on OCed Q6600. So performance increase is quite apparent and worth it for this class of tasks.