shareVM- Share insights about using VM's

Simplify the use of virtualization in everyday life

Posts Tagged ‘virtualbox vdi

Best Practice: Defrag VMDK, VHD, VirtualBox Virtual Disk

leave a comment »

Wikipedia describes defragmentation as

a process that reduces the amount of fragmentation in file systems. It does this by physically organizing the contents of the disk to store the pieces of each file close together and contiguously. It also attempts to create larger regions of free space using compaction to impede the return of fragmentation.

Generically, the defragmentation of a Windows guest within a virtual disk running on a Windows host (Windows on Windows) requires a three-step process:

  1. Defragment the guest
  2. Defragment the virtual disk
  3. Defragment the host

On a Linux host or guest, the ext3 and ext4 file systems are more resilient to defragmentation.

Windows on Windows

You should perform the following steps whether you are using a Microsoft VHD, VirtualBox VDI or VMware VMDK virtual disk,

  1. On a Windows guest OS, run the Windows Disk Defragmenter to defragment the files within the volumes stored inside the virtual disk.
  2. Next, power down the virtual machine and defragment the virtual disk using contig. Defragmenting the virtual disk simply reorganizes the blocks so that used blocks move towards lower-numbered sectors and unused blocks move towards higher-numbered sectors.
  3. Run the Windows Disk Defragmenter to achieve an overall defragmentation of all files on the host including the virtual disk.

VMware VMDK specific

The following steps can be used generically for VMware VMDK, for Windows on WIndows or any other suppoted platforms. vmware-vdiskmanger:is a standalone tool for defragmenting a growable VMware Workstation, VMware Fusion or VMware Server, vmdk when it is offline. Note that you cannot defragment:

  • Preallocated virtual disks
  • Physical hard drives
  • Virtual disks that are associated with snapshots.

The recommended steps for defragmenting a vmdk are:

  1. On a Windows guest OS, run the Windows Disk Defragmenter to defragment the files within the volumes stored inside the VMDK.
  2. Next, power down the virtual machine and defragment the vmdk using the command vmware-vdiskmanager -d myVirtualDisk.vmdk. Defragmenting the vmdk simply reorganizes the blocks so that used blocks move towards lower-numbered sectors and unused blocks move towards higher-numbered sectors.
  3. If the host OS is also Windows, run the Windows Disk Defragmenter to achieve an overall defragmentation of all files on the host including the VMDK.

Why do Windows C drives get full in virtual disks?

leave a comment »

A real life experience posted by a member in the VMware vCenter Server Communities yesterday (Feb 8, 2010):

I have installed vc with sql 2005 express, now my vCenter server c:\ is almost full
is it possible to move my vCenter database to another drive

The solution recommended by an expert is:

you can install a new server with more space and migrate the data as following.
link to kb post
But you can use also tolls like gparted or dell_expart to incrase your space.

While this recommendation is consistent with the perceived state of the art, it does have the following impact:

It is not going to affect the running VMs and also ESX but you/VSC may see a disconnect for a while.

Another member recommends a different approach

A different approach would be to extend the c-drive.
We have recently released a tool (fatVM) to make this easy (or easier).
It creates the extended VM in a new directory (with the original as parent). Does not touch the original files. Is able to extend most VM in a couple of minutes.
Here is the link: http://www.gudgud.com/fatvm

A third member is contemplating a similar move:

I have a 4 host ESX 3.5U4 system.My VCenter is pointing to an external SQL server. I am about to upgrade to vSphere and want to have the SQL running on on the VCenter server itself – most likely using SQL Express. I have the same concern about space.

You must have noticed the pattern that is emerging. Your C:drive can get full when you are using a database system, or a log aggregation server, within a VM that has a pre-allocated disk and size of the data is growing. As a best practice, review your apps for potential of data growth before pre-allocating the size of the VM.

Virtualbox branching / branched snapshots

leave a comment »

An illustrated, step-by-step procedure for branching snapshots using Virtualbox 3.1.0 is available here.

Roman Kennke’s post on branching snapshots describes the feature and provides a how-to guide: 

A typical use case would be to install an OS into a virtual disk, make that virtual disk read-only and use it as base image for several branches.

  1. For example, in one branch I would do testing/debugging of stuff that I develop. There might be several branches I use for testing.
  2. Then I might need a branch in which I install a build environment for OpenJDK, which could in turn be used for several more sub-branches for OpenJDK6 builds and OpenJDK7 builds.
  3. In another branch off the base image I would run tax software. Etc

Written by paule1s

December 10, 2009 at 8:27 pm

Posted in virtualbox

Tagged with , ,

Top 3 formats for compressing Virtual Appliances

with 2 comments

Total VM’s on VMWare Virtual Appliance Marketplace : 2560
Number of distinct download sites providing these appliances : 674

The following table lists the compression formats used for 408 appliances we have examined:

compression-formats

Amongst the compression formats, zip is the most popular one (42%), with 7z the next in sequence (23%) for Windows; gz is the most popular one for Linux (18%)

We intend to publish more statistics over the next few days. Watch this space.

Written by paule1s

February 5, 2009 at 11:05 pm

Who is feeling the pain for shrinking VM’s?

with one comment

From informal surveys conducted with ex-colleagues and developers in other companies, and the inquiries we are receiving, the picture that has emerged in my mind is that shrinking VM’s is a need for

  • Developers who can store a limited number of VMs on their personal workstations/laptops.
  • Developers and particularly System Administrators who are concerned about the performance of their app within the VM at run time
  • Release Engineers/System Administratrators who have to transfer VM images from a staging/build/test machine to one or more production servers in a remote data center, or in the cloud, and are interested in reducing their size, and hence the elapsed time for the file transfer .

Release Engineers/System Administrators dealing with production VMWare ESX deployments seem to be storing VM’s on storage that supports de-duplication, primarily from

Development teams in large companies do use NetApp filers to provide shared NFS-mounted storage for development teams. However, I am not sure whether these filers support de-duplication, and if they do, whether it is enabled. If de-duplication is indeed enabled, then these shared filesystems typically act as passive storage for VM’s created by developers/build masters and the overall demand for storage reduces considerably.

However, there are development teams that also use commodity storage for file shares. If they are Linux file shares, then the ext3 file system seems to be the industry-standard; otherwise, they are Windows NTFS file shares. Both these file systems are also found on developer desktops – they don’t support de-duplication and have the exact same defragmentation and VM shrinking challenges that the developer faces on his personal workstation – except that the pain for managing file shares is felt by the IT admin/operator responsible for its well being. (It is worth reading this informative article about dealing with VM snapshots and linnked clones on Windows NTFS).

The major pain for managing several VM’s stored on commodity storage on a workstation/laptop seems to be felt by the developer since he can only offload older VM’s on a USB drive or a shared filesystem for “archival”, and has to keep the current one’s that he is testing on the local disk. We are hoping this blog is providing pointers to the “how to”  for the developers, release engineers and system adminsitrators.

Written by paule1s

January 7, 2009 at 7:31 pm

Shrinking VM snapshots of builds and releases

with 2 comments

The traffic on our blog tells us that there is intense interest in the topic of shrinking virtual machines – vmdk’s vhd’s, vdi’s, to recover free space and improve performance. Our offer to release a software tool to let you view how much storage you could recover has brought several responses from people who are willing to try it.

I found the following response very interesting because it outlines the development usage of VM’s, in contrast with production scenarios that one usually reads about, within a very large multinational software development company whose products are used worldwide:

We capture the build environment and its dependencies together with the end product of the build, the released software, in a VM. During the lifecyle of a major product release, we end up with over 300 builds, i.e., over 300 VM’s. Some of these VM’s are shared with other development teams who have to integration test their builds with our own. Builds corresponding to the major external milestones, e.g., Beta-1, Beta-2, etc. are provided to our technology partners and even to end customers so that we can get early feedback on our product.

We store all these VM’s on a Windows file share and we preserve all VM’s throughout the release cycle and those corresponding to major internal integration and external Beta milestones for at least 6 months beyond the release date. Our goal is to be able to recreate an issue and verify that it is indeed fixed in a later release. So our storage needs are growing annually and its demands on our budget are rising in prominence. We would love to be able to shrink these images to the minimum essential size and leverage our investment in storage far more effectively.

Our Release Management team uses spreadsheets and file naming conventions for managing this store. However, at any given point in time it is very hard for us to match issues that are reported precisely to the release (VM) where it was introduced.

We do share these VM’s between teams but the sharing is achieved using file shares on the LAN. This is very limiting for us since we have development centers in the US (East and West Coast), China, India and sales engineers globally, but given the network bandwidth demands, we don’t even attempt large transfers. Given the size of these images, we support http downloads from an intranet site for teams that are not on our LAN.

In summary, this development team is facing three problems:

  1. VM compaction to recover free space and reduce impact on their budget
  2. VM tagging to identify them by key attributes
  3. VM network transfers take very long to complete

Thank you for sharing this kind of input, we intend to help you resolve such issues.

Written by paule1s

January 5, 2009 at 2:54 pm

VirtualBox – setup, share, shrink, convert

with 3 comments

Written by paule1s

December 17, 2008 at 4:03 pm