shareVM- Share insights about using VM's

Simplify the use of virtualization in everyday life

Posts Tagged ‘defrag

Best Practice: Defrag VMDK, VHD, VirtualBox Virtual Disk

leave a comment »

Wikipedia describes defragmentation as

a process that reduces the amount of fragmentation in file systems. It does this by physically organizing the contents of the disk to store the pieces of each file close together and contiguously. It also attempts to create larger regions of free space using compaction to impede the return of fragmentation.

Generically, the defragmentation of a Windows guest within a virtual disk running on a Windows host (Windows on Windows) requires a three-step process:

  1. Defragment the guest
  2. Defragment the virtual disk
  3. Defragment the host

On a Linux host or guest, the ext3 and ext4 file systems are more resilient to defragmentation.

Windows on Windows

You should perform the following steps whether you are using a Microsoft VHD, VirtualBox VDI or VMware VMDK virtual disk,

  1. On a Windows guest OS, run the Windows Disk Defragmenter to defragment the files within the volumes stored inside the virtual disk.
  2. Next, power down the virtual machine and defragment the virtual disk using contig. Defragmenting the virtual disk simply reorganizes the blocks so that used blocks move towards lower-numbered sectors and unused blocks move towards higher-numbered sectors.
  3. Run the Windows Disk Defragmenter to achieve an overall defragmentation of all files on the host including the virtual disk.

VMware VMDK specific

The following steps can be used generically for VMware VMDK, for Windows on WIndows or any other suppoted platforms. vmware-vdiskmanger:is a standalone tool for defragmenting a growable VMware Workstation, VMware Fusion or VMware Server, vmdk when it is offline. Note that you cannot defragment:

  • Preallocated virtual disks
  • Physical hard drives
  • Virtual disks that are associated with snapshots.

The recommended steps for defragmenting a vmdk are:

  1. On a Windows guest OS, run the Windows Disk Defragmenter to defragment the files within the volumes stored inside the VMDK.
  2. Next, power down the virtual machine and defragment the vmdk using the command vmware-vdiskmanager -d myVirtualDisk.vmdk. Defragmenting the vmdk simply reorganizes the blocks so that used blocks move towards lower-numbered sectors and unused blocks move towards higher-numbered sectors.
  3. If the host OS is also Windows, run the Windows Disk Defragmenter to achieve an overall defragmentation of all files on the host including the VMDK.

Should you de-fragment Virtual Disks?

with one comment

Windows de-fragmentation tool or some other commercial alternative, need 5-15% of free disk space, for the tool to be effective. Sometimes it may need more if you have some very large files (like video or database files). Below is the layout of c-drive of may virtual machine. The red segments you see are the fragmented files.

If you have a file with one large segment, for the defrag to be effective it has to move this segment to a free area and copy the rest of the segments with it to make the file contiguous. If there is no place to copy the large extent of a file, then it wont get defragmented.

The best way to de-fragment is to get an empty disk and copy all the files onto the empty disk. So the more free disk you have the better these tools will perform.

Also how you think about de-fragmentation in a virtual disk is very different than how we think about de-fragmentation in a physical world. Take the above disk it is a virtual disk 2GB Max Extent Sparse

The disk was full and then I extended the disk (with fatVM) and then defragmented one file (you can do that with Mark Russinovich’s Contig Tool http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx). You can see that the files are contiguous (blue) in the extended portion. The original disk clearly requires defragmentation, but without extending it, we would not have been able to get the key database file to be contiguous.

It makes one ask the question whether you really need the traditional way of defrag the virtual disk. It is much faster to extend the disk and/or attach a separate disk and simply copy over all the files and re-place the original disk with the new extended disk.

Another advantage of doing this is that it is much faster than defragging also you can improve the performance of the virtual machine considerably. Also you can take the files which are static (don’t change)  by taking the files in a virtual machine which don’t change and making the base new disk for c-drive a flat file instead of a sparse disk as the sparse disk is not really saving you anything once you get full. If you have a parent which is flat and then a child which is sparse you get the best of both worlds.

In my limited experience instead of defrag, do the following

  • create a new flat disk, copy all the files from C: to the new disk
  • make the new disk your c: drive
  • create a clone of the base disk (which by definition is sparse)
  • extend the sparse disk

Your virtual machine’s performance will be significantly improved.

Thin Provisioning – when to use, benefits and challenges

with 4 comments

There are excellent posts by two prominent authors that provide a lot of insight into the nuances of using thick or thin provisioning for VM’s: Thin Provisioning Part 1 – The Basics and Thin Provisioning Part 2 – Going Beyond by Vaughn Stewart of NetApp and Thin on Thin – where should you do Thin Provisioning by Chad Sakac of EMC.

Synopsis:
Escalating storage costs are stalling the deployment of virtualized data centers and it is becoming increasingly important for customers to leverage storage technology developed by VMware and its storage partners, Netapp and EMC for reducing storage costs.

vmdk formats:

vmdk formats

VMFS blocks
pre-allocated

Disk array block
pre-allocated

Disk array blocks
pre-allocated

Thin

No

No

No

Thick (Non-zeroed)

Yes

No

No

Eager zeroed thick

Yes

Yes

Yes

 

Recommendations:
Use Thin on Thin (Thin vmdk’s and Thin Provisioning on the storage array) for the best storage utilization because they allocate storage capacity from the datastore and storage array only on demand.

Stewart:

The Goal of Thin Provisioning is Datastore Oversubscription  The challenge is that datastore, and all of its components (VMFS, LUNs, etc…) are static in terms of storage capacity. While the capacity of a datastore can be increased on the fly, this process is not automated or policy driven. Should an oversubscribed datastore encounter an out of space condition, all of the running VMs will become unavailable to the end user. In these scenarios the VMs don’t ‘crash’ the ‘pause’; however, applications running inside of VMs may fail if the out of space condition isn’t addressed in a relatively short period of time. For example Oracle databases will remain active for 180 seconds, after that time has elapsed the database will fail.

Sakac:

If you DO use Thin on Thin, use VMware or 3rd party usage reports in conjunction with array-level reports, and set thresholds with notification and automated action on both the VMware layer (and the array level (if you array supports that). Why? Thin provisioning needs to carefully manage for “out of space” conditions, since you are oversubscribing an asset which has no backdoor (unlike how VMware oversubscribes guest memory which can use VM swap if needed). When you use Thin on Thin – this can be very efficient, but can “accelerate” the transition to oversubscription.

Sakac:

The eagerzeroedthick virtual disk format is required for VMware Fault Tolerant VMs on VMFS (if they are thin, conversion occurs automatically as the VMware Fault Tolerant feature is enabled). It continues to also be mandatory for Microsoft clusters (refer to KB article) and recommended in the highest I/O workload Virtual Machines, where the slight latency and additional I/O created by the “zeroing” that occurs as part and parcel of virtual machine I/O to new blocks is unacceptable.

vmdk growth:

Stewart:

VMDK grew beyond the capacity of the data which it is storing. The reason for this phenomenon is deleted data is stored in the GOS file system. When data is deleted the actual process merely removes the content from the active file system table and marks the blocks as available to be overwritten. The data still resides in the file system and thus in the virtual disk. This is why you can purchase undelete tools like WinUndelete.

Don’t run defrag within a thin provisioned VM

Stewart:

the defragmentation process results in the rewriting all of the data within a VMDK. This operation can cause a considerable expansion in the size of the virtual disk, costing you your storage savings.

How to recover storage

Stewart:

First is to zero out the ‘free’ blocks within in the GOS file system. This can be accomplished by using the ‘shrink disk’ feature within VMTools or with tools like sdelete from Microsoft. The second half, or phase in this process, is to use Storage VMotion to migrate the VMDK to a new datastore.

The second half, or phase in this process, is to use Storage VMotion to migrate the VMDK to a new datastore. You should note that this process is manual; however, Mike Laverick has posted the following guide which includes how to automate some of the components in this process. Duncan Epping has also covered automating parts of this process.

A year in review: What are our readers looking for?

leave a comment »

Our readers are primarily asking questions like:

  • How can I free up disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
  • Where can I find the best virtual appliances/ Top 10 virtual appliances?
  • How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
  • Who are the competitors for ec2?

An analysis of the search terms shows interesting clusters:

Serial

Topic

% of queries

Search terms

1

ext4 defragmentation

23%

ext4 defrag, defrag ext4, ext4 defragment, defragment ext4

2

ubuntu ext4 defragmentation

14%

ext4 defrag ubuntu, ext4 ubuntu defrag, ubuntu ext4 defrag, ubuntu defrag ext4, defrag ext4 ubuntu, defrag ubuntu ext4

3

vmware virtual appliance

14%

vmware virtual appliance, vmware virtual appliances, top vmware appliances, top 10 vmware appliances, best vmware appliances

4

virtual appliance

5%

virtual appliance, virtual appliances, top appliances, top 10 appliances, best appliances

5

vmware firewall appliance

5%

vmware firewall appliance, vmware appliance firewall

6

ubuntu defragmentation

4%

defrag ubuntu, ubuntu defrag, defragment ubuntu, ubuntu defragment

7

ec2 competitors

4%

amazon ec2 competitors, ec2 competitors

8

windows 7 virtual appliance

4%

windows 7 virtual appliance, virtual applaince windows 7

9

ext3 defragmentation

4%

ext3 defrag, defrag ext3, ext3 defragment, defragment ext3

10

convert vdi to vhd

3%

convert vdi to vhd, vdi to vhd

If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.

Thank you. I appreciate your interest in this blog.

Top 10 Posts for Q1 2009

leave a comment »

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

  1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
  2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
  3. VirtualBox – setup, share, shrink, convert (842)
  4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
  5. How does shrink with vmware disk manager work? (614)
  6. Most popular VMWare Virtual Appliances for Security (607)
  7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
  8. Most popular VMWare Virtual Appliances for Web Apps (558)
  9. Virtual Machine Disk Image Compression (320)
  10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

  • ext4 defrag ubuntu
  • ext4 defrag
  • convert vdi to vhd
  • e4defrag ubuntu
  • virtualbox shrink
  • rsync vmdk
  • wubi
  • defrag ubuntu
  • defrag ext3
  • windows 7 virtual appliance
  • defragment ext3
  • vmware appliances
  • defrag ext4
  • xen vhd
  • ubuntu ext4 defrag
  • defrag ext4 ubuntu
  • vmware firewall appliance
  • vmware appliance
  • “vdi to vhd”
  • convert vhd to xen
  • ext3 defrag
  • windows 7 beta vmware virtual appliances
  • defrag fedora
  • ext3 defragmentation
  • virtual appliance windows 7
  • ubuntu defrag
  • hercules load balancer virtual appliance
  • fedora defrag
  • convert vmdk to xen
  • shrink vmware disk

Top 10 referrers for Q1 2009

leave a comment »

Top 12 referrers over the past 3 months

leave a comment »