shareVM- Share insights about using VM's

Simplify the use of virtualization in everyday life

Should you de-fragment Virtual Disks?

with one comment

Windows de-fragmentation tool or some other commercial alternative, need 5-15% of free disk space, for the tool to be effective. Sometimes it may need more if you have some very large files (like video or database files). Below is the layout of c-drive of may virtual machine. The red segments you see are the fragmented files.

If you have a file with one large segment, for the defrag to be effective it has to move this segment to a free area and copy the rest of the segments with it to make the file contiguous. If there is no place to copy the large extent of a file, then it wont get defragmented.

The best way to de-fragment is to get an empty disk and copy all the files onto the empty disk. So the more free disk you have the better these tools will perform.

Also how you think about de-fragmentation in a virtual disk is very different than how we think about de-fragmentation in a physical world. Take the above disk it is a virtual disk 2GB Max Extent Sparse

The disk was full and then I extended the disk (with fatVM) and then defragmented one file (you can do that with Mark Russinovich’s Contig Tool http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx). You can see that the files are contiguous (blue) in the extended portion. The original disk clearly requires defragmentation, but without extending it, we would not have been able to get the key database file to be contiguous.

It makes one ask the question whether you really need the traditional way of defrag the virtual disk. It is much faster to extend the disk and/or attach a separate disk and simply copy over all the files and re-place the original disk with the new extended disk.

Another advantage of doing this is that it is much faster than defragging also you can improve the performance of the virtual machine considerably. Also you can take the files which are static (don’t change)  by taking the files in a virtual machine which don’t change and making the base new disk for c-drive a flat file instead of a sparse disk as the sparse disk is not really saving you anything once you get full. If you have a parent which is flat and then a child which is sparse you get the best of both worlds.

In my limited experience instead of defrag, do the following

  • create a new flat disk, copy all the files from C: to the new disk
  • make the new disk your c: drive
  • create a clone of the base disk (which by definition is sparse)
  • extend the sparse disk

Your virtual machine’s performance will be significantly improved.

Why do Windows C drives get full in virtual disks?

leave a comment »

A real life experience posted by a member in the VMware vCenter Server Communities yesterday (Feb 8, 2010):

I have installed vc with sql 2005 express, now my vCenter server c:\ is almost full
is it possible to move my vCenter database to another drive

The solution recommended by an expert is:

you can install a new server with more space and migrate the data as following.
link to kb post
But you can use also tolls like gparted or dell_expart to incrase your space.

While this recommendation is consistent with the perceived state of the art, it does have the following impact:

It is not going to affect the running VMs and also ESX but you/VSC may see a disconnect for a while.

Another member recommends a different approach

A different approach would be to extend the c-drive.
We have recently released a tool (fatVM) to make this easy (or easier).
It creates the extended VM in a new directory (with the original as parent). Does not touch the original files. Is able to extend most VM in a couple of minutes.
Here is the link: http://www.gudgud.com/fatvm

A third member is contemplating a similar move:

I have a 4 host ESX 3.5U4 system.My VCenter is pointing to an external SQL server. I am about to upgrade to vSphere and want to have the SQL running on on the VCenter server itself – most likely using SQL Express. I have the same concern about space.

You must have noticed the pattern that is emerging. Your C:drive can get full when you are using a database system, or a log aggregation server, within a VM that has a pre-allocated disk and size of the data is growing. As a best practice, review your apps for potential of data growth before pre-allocating the size of the VM.

Download fatVM for VMware virtual disk single-click extension

leave a comment »

We are pleased to announce that fatVM Beta for VMware Fusion and VMware Workstation is available for download.

fatVM is a reliable, robust, and safe, single click solution for extending the C drive of your virtual disk that is becoming full. fatVM extends the VM even in cases when it has snapshots and clones. You can use fatVM with full confidence because it preserves your original disk, which remains available to you in case the need ever arises.

Mac-friendly

A Mac user can download and install the fatVM dmg just like any other Mac application. fatVM provides a simple, intuitive, interface and a reliable process that hides the technical complexity of extending a virtual disk.

For most Mac users, extending the C drive of Window® running within VMware Fusion® can be a daunting task. There are several online tutorials that do provide instructions on how to perform low-level Windows system administration tasks, invoke UNIX utilities and manage disk formats (like partition tables) using third-party tools. Even if you were to follow such steps, they do not handle dependencies, e.g., snapshots or clones, well and you may not necessarily be able to extend the disk. You may also discover that recovering from failures to extend the C-drive can be a very challenging experience.

fatVM packages the process of extending the VM and provides you with the capability of executing it through a single click.

Windows-friendly

A Windows user can download and install the fatVM.exe just like any other Windows application. fatVM works with VMware Workstation in a similar fashion as described for VMware Fusion.

Single Click Operation

fatvm shows you a menu of VM’s that belong to VMware Fusion’s ® Virtual Machine Library. You can either select a VM from that menu, or you can you can drag and drop the .vmx configuration file for a new VM, to extend it. fatVM analyzes the selected VM and discovers the current size of the VM’s disk. All you have to do is to select the size you want to extend it to and press Extend, fatVM does the rest:

  1. Verify whether VM is running and terminate the operation if it is found running
  2. Analyze VM to determine its current size
  3. Discover partitions and system disk
  4. Verify disk type
  5. Create snapshot and new disk
  6. Attach ISO
  7. Create Windows partition
  8. Extend Windows file system
  9. Boot Windows to ensure that chkdsk runs
  10. Complete VM extension.

Super Fast Cross-OS VM Browser
Just like Google’s Chrome, a fast, lightweight, cross-OS Web browser, fatVM comes bundled with a super fast, lightweight browser for offline VM’s. You can view 50+ VM’s simultaneously. Just drag and drop the VM you would like to explore. Some highlights of the browser are:

  • Supports cross-OS Browsing: Browse Linux VM’s on Windows, Linux and Windows VM on Mac
  • Discovers Applications installed inside the VM’s. Supports Ubuntu, Red Hat, and Windows Virtual Machines
  • Runs in user mode, does not install any kernel level components
  • Includes Offline Registry & Partition Table Browser for Windows VM’s

Undo and Rollback

fatVM leaves the original files untouched and creates an extended VM in a separate directory.

  • If there is any problem with extending the VM, you can select Rollback under Tools and simply delete the fatVM folder to leave your work environment in the same state as before the extension was attempted.
  • The extended VM is clone of the original VM. For this reason, the original VM must not be booted after it has been extended.

Written by paule1s

February 7, 2010 at 4:11 pm

Posted in extend virtual disk

Tagged with , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Survey: Extend / Expand Microsoft Hyper-V VHD

with one comment

Before you begin, create a backup copy of the virtual disk so that you can restore it in case of failures. Extending a virtual disk can be a complex operations for several reasons, e.g.,:

  • The virtual disk that is full happens to contain the boot/primary partition for Windows
  • There are snapshots
  • There is insufficient disk space available on the physical drive to permit growth
  • It is a multi step process and involves use of different tools and commands in a specific sequence.

You’ll find several posts regarding this topic on Google. Essentially, they are all variations of the following posts:

  1. Tutorial to extend boot/system partition using EASEUS Partition Master for dynamic and fixed VHD’s
  2. VHD Utility can be used for extending file system partitions for dynamic and fixed VHD’s.
  3. Microsoft is offering a free VHD tool, including its source code, which is an unmanaged code command-line tool that provides useful functions, e.g.,
  4. Create a new 10 GB fixed VHD:
    VhdTool.exe /create "c:\Program Files\MyApp\foo.vhd" 10737418240

    Convert an existing flat image file into a VHD & do not output status to the command line. VhdTool.exe /convert bar.img /quiet

    Extend an existing fixed format VHD to a larger size.
    VhdTool.exe /extend foo.vhd 21474836480

    Repair a Hyper-V snapshot chain broken by expanding the base VHD.
    VhdTool.exe /repair base.vhd base_EF2F9402-E85B-402F-A979-631CB287C2C4.avhd

  5. Tutorial for resizing a virtual disk using VHD Resizer
  6. Expanding Virtual Hard Disks with Hyper-V

Survey: Extend / Expand VMware virtual disk

with 3 comments

Before you begin, create a backup copy of the virtual disk so that you can restore it in case of failures. Extending a virtual disk can be a complex operations for several reasons, e.g.,:

  • The virtual disk that is full happens to contain the boot/primary partition for Windows
  • There are snapshots
  • There is insufficient disk space available on the physical drive to permit growth
  • It is a multi step process and involves use of different tools and commands in a specific sequence. Creative individuals have devised workarounds to extend the primary partition for Windows by using Linux boot disks and disk partitioning tools such as GParted, and you have to get this right.

A blog post that expands on each of the above is here.

You’ll find several posts regarding this topic on Google. Essentially, they are all variations of the following posts

VMware Workstation or Fusion

How To Extend Virtual Machine Hard Disk (VMware) by Kalpesh Prajapati and Extend Boot Volume on Windows Server 2000/2003 by Dominic Rivera provide an illustrated tutorial that demonstrates the extension of Windows 2003 guest.  You will notice that there are several steps and from reviewing the comments posted for the former, you’ll realize that these steps have not worked for its readers uniformly.

Expanding a drive within a  VMware image by Sean Deasy outlines two methods

  1. Extend the non-system/data drive using WIndows tools. This approach preserves snapshots
  2. Extend the system drive using vmware-vdiskmanager tools. However, this approach causes loss of snapshots

VMware Server

Adding a new disk to a VMware virtual machine in Linux by Matt Topper provides an illustrated step-by-step how-to gude

ESX

A recent authoritative and comprehensive post about Re-sizing Virtual Disks by Eric Seibert is a great starting point. It provides step-by-step how-to guide

  1. Using vmkfstools and GParted to extend a disk
  2. Using VMware Converter to shrink or extend a disk
  3. Using vmkfstools and another Windows virtual machine to extend a disk
  4. Using vmkfstools and System Rescue CD to extend a disk
  5. Using Knoppix Live CD with QtPartEd to shrink a disk
  6. Using Ghost or another 3rd party imaging product to shrink a disk

Entrepreneurs and Startups: 10 Lessons in Entrepreneurship

with one comment

Om Malik’s post The Essential Startup Reader: 10 Lessons in Entrepreneurship is a collection of insightful essays about what makes entrepreneurs and startups tick. It is a must read.

Written by paule1s

January 8, 2010 at 1:03 am

how “capitalism” forces virtualization downstream

leave a comment »

Over the last decade we have systematically added a layer of indirection at every interface in the stack. These days we call this virtualization!

On a NetApp Filer we had

> raid disk group > volume

The problem was you could not expand/shrink Raid Groups on the fly, you couldn’t move data easily between different Raid Groups. We get a layer of indirection or virtualization

> raid disk group > aggregate > flex volume>

Since an aggregate was logical instead of physical, it could be expanded or shrunk without changing the volume, you could move data around.

On a USB Disk

If we look inside the disk itself, especially usb flash devices we went from

cylinder, heads, sector > logical table > device abstraction

Again this allowed the rotation of different logical sectors to different physical cells, to ensure a single cell was not rewritten more times than its lifetime.

In a SAN

we put a switch between the Raid Groups and the computer. The switch puts a layer of indirection between the blocks and the computer

You knew all that :-). So what does it have to do with capitalism. My simplistic definition of capitalism is that the system will remove all inefficiencies in a chain and who ever will remove them stands to benefit economically. Or said another way: money finds its way into the right pockets!

So look at the stack today:

chips > motherboard, network, storage, bios > hypervisor > OS > Security, Backup etc > Business, Productivity Apps

Every layer presents an interface to the layer above. Each layer is also owned by different companies in the eco-system. Each of those companies has pressure to maximize its revenue. Tasked with this difficult challenge, you look at the layer above and see what is selling and can you add it to your layer. Happens naturally over time: intel added virtulization support, phoenix bios is adding the hypervisor, operating systems are trying to add backup and security …. The cycle goes on ….

Virtualization will be “innovated” always in a higher layer of the stack and commoditized by the lower layers.

The higher layer in the stack finds a lot of new functionality and benefit by making interface to a lower layer “logical”. They take this to market, till at some point the lower layer realizes that this is their API, they should move virtualization into their layer. The pressure to do this is extreme and the time frame to monetize this really small:

  • Imagine the tussle between VMW and the storage vendors. VMW introduces logical disks with cloning, but storage vendors want to offer logical luns and volumes and disk files, as this moves the cloning functionality from the hypervisor to the storage.
  • Imagine: Western Digital or Seagate could create multiple disks (vhd/vmdk files) on a single physical disk and then offer the capabilities to grow, shrink, move data between them. Even add networking to the disk controller, then different disks can connect to each other. They can do that if the processing power, memory reach a price point that it can be embedded directly into the component or lower layer. Which is what effectively happened to computing.
  • VMW introduces logical network switch, Cisco jumps in with nexus-V

For a consumer this is a good thing, but money and value are shifting down stack across different companies, which have to co-exist in the eco-system (cisco, intel, emc, vmw), yet guard their innovation from becoming commoditized.

Written by RS

December 13, 2009 at 8:36 am