shareVM- Share insights about using VM's

Simplify the use of virtualization in everyday life

Posts Tagged ‘sparse file

How much will the VM shrink?

with one comment

Shrinking a VM today is a time consuming process: zero out the free blocks, defrag, use vmware’s tools to shrink it partition by parition. I wanted a tool which would tell me after doing all that how much would it help. This turned out to be a complicated process as you have to find the free clusters in NTFS and find out whether they are actually allocated as in sparse VMDK files all blocks may not be allocated. I ran the tool on some VMs see below: Windows 2003, Windows NT & a couple of XP VMs.

fcp -f c:\work\vhd\w2k3.vhd
NTFS Free Sectors 594632
Free Sectors Allocated in Virtual Image 294496
Maximum Possible Saving by Shrinking 143 MB

fcp -f “c:\work\vm\w1\Windows XP Professional.vmdk”
NTFS Free Sectors 1049160
Free Sectors Allocated in Virtual Image 883144
Maximum Possible Saving by Shrinking 431 MB

fcp -f “c:\work\vhd\Windows XP Hard Disk.vhd”
NTFS Free Sectors 30124176
Free Sectors Allocated in Virtual Image 847013
Maximum Possible Saving by Shrinking 413 MB

fcp -f “c:\work\vm\wnta\wnta.vmdk”
NTFS Free Sectors 27724512
Free Sectors Allocated in Virtual Image 71656
Maximum Possible Saving by Shrinking 34 MB

I am thinking of releasing the tool, it only supports vmdk and vhd files today with NTFS file system. If you would like to get an early release, would be thrilled to share let me know.

Written by RS

December 31, 2008 at 8:21 am

Virtual Machine Disk Image Compression

with 2 comments

Experience with running out of storage

Background about sparse files, Windows NTFS and VHD

VHD Pre-Compactor & Compactor

When you create a new VM you can pre-allocate all the storage in one shot or let the storage grow on demand. In the latter case, the VM uses a sparse file

VMDK Wipe & Shrink

3rd Party (Not from MSFT & VMW)

Rsync: for transferring files between machines

Research

How does shrink with vmware disk manager work?

with 3 comments

I found the explanation below on the invirtus site to be the simplest. One of the side-effects of shrinking VMDK files with this approach is that with sparse files it will actually increase the size of the file as the filler files are created and then decrease it. So if you are very close to full capacity on your hard drive this may not be the best option.

VMware® disk types are files with the extension .vmdk (dot vmdk). Shrinking vmdk files (shrink vmdk) can be accomplished with VMware’s built-in VMware Tools. VMware Tools has a shrink facility built-in which can be used on a dynamically expanding disk type (disk space is not pre-allocated for this virtual disk type and can be reclaimed). To do so go to the Windows Control Panel, open VMware Tools then click the Shrink tab.

Shrinking is a two-step procedure.

Step 1: When you click “Prepare to shrink” the tools utility launches a program which fills the free-space on the entire disk with zero filled files named wiper0, wiper1, wiper2, etc, until the entire disk is filled. These “wiper” files are then deleted at the conclusion of the process and the free-space is reclaimed.

Step 2: At the conclusion of shrink preparation you will be presented with a dialog which asks if you would like to shrink your disk(s). If you answer yes to this prompt VMware Tools essentially creates a new disk with the same partition characteristics as the existing disk and simply writes contents of the existing disk into the new disk and at the conclusion of the process it deletes the old disk, renames the new disk and makes it current.

VM Optimizer relies on step 2 for its final procedure but adds hundreds of additional methods to the first step. The result is that when shrinking your vmdk file VM Optimizer will give you the smallest possible virtual machine.

Written by RS

December 12, 2008 at 3:19 pm

Ubuntu VMDK Fragmentation and Vmware allocation size?

leave a comment »

I run vmware workstation 6.1 on my laptop and use a ubuntu VM in it. The vmdk is setup to use sparse files with max extent 2GB (vmdk file described below). I started with the ubuntu server which is very minimal and then installed ubuntu-xdesktop on it. The xdesktop package installed close to a 1000 packages and expanded the sparse disk a lot. Here is a result of running windows defrag (start–>programs–>accessories–>system tools–>disk defragmenter)

Fragments  File Size       Most fragmented files

202             1.56 GB         workvmu1u1-s001.vmdk

So it has 202 fragments. 1.56GB/202 ~ 84MB.

The logical allocation units can be

– grain: of the grainsize = 128 sectors or 64KB.

– grain table: Pointers to 512 grains are stored in a grain table. thus the grain table covers about 32Mb. Looks like it would allocate 2 graintable?

It is possible it allocates a grain-table worth of data at one time (although may not be used for the same grain table) and the underlying OS as it got more fragmented could have split single allocations into more than one fragments.

Technorati Tags: ,,,

Complete Fragmentation Report

Volume (C:)
Volume size                                = 74.46 GB
Cluster size                               = 4 KB
Used space                                 = 60.42 GB
Free space                                 = 14.04 GB
Percent free space                         = 18 %

Volume fragmentation
Total fragmentation                        = 19 %
File fragmentation                         = 38 %
Free space fragmentation                   = 0 %

File fragmentation
Total files                                = 332,864
Average file size                          = 293 KB
Total fragmented files                     = 11,312
Total excess fragments                     = 57,361
Average fragments per file                 = 1.17

Pagefile fragmentation
Pagefile size                              = 2.00 GB
Total fragments                            = 1

Folder fragmentation
Total folders                              = 78,930
Fragmented folders                         = 2,309
Excess folder fragments                    = 8,623

Master File Table (MFT) fragmentation
Total MFT size                             = 416 MB
MFT record count                           = 412,420
Percent MFT in use                         = 96 %
Total MFT fragments                        = 3

——————————————————————————–
Fragments       File Size       Most fragmented files
3,542           555 MB          System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP360A0064665.exe
1,791           214 MB          workdownload7.0-RELEASE-i386-livefs.iso
1,768           695 MB          workdownload7.0-RELEASE-i386-disc2.iso
1,022           509 MB          workdownload7.0-RELEASE-i386-disc1.iso
849             8 MB            Documents and SettingsrosenLocal SettingsTempdd_NET_Framework20_Setup02DE.txt
790             352 MB          workdownload7.0-RELEASE-i386-disc3.iso
470             237 MB          workdownload7.0-RELEASE-i386-docs.iso
440             27 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP344A0062479.reg
347             22 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP377A0070304.old
329             21 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP343A0062133.reg
291             1 KB            WINDOWSsystem32configsystem.LOG
285             3 MB            Program FilesMicrosoft SQL Server100Setup BootstrapLogVSExpress_9.0sql_engine_core_shared_Cpu32_1.log
263             16 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP344A0062478.reg
252             16 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP343A0062132.reg
247             15 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP359A0064513.exe
218             14 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP359A0064400.rbf
202             1.56 GB         workvmu1u1-s001.vmdk
182             11 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP384A0071645.dll
182             13 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP349A0063506.rbf
179             20 MB           WINDOWSpfirewall.log
174             11 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP343A0061844.rbf

u1.vmdk descriptor file

# Disk DescriptorFile
version=1
encoding=”windows-1252″
CID=d23aac87
parentCID=ffffffff
createType=”twoGbMaxExtentSparse”

# Extent description
RW 4192256 SPARSE “u1-s001.vmdk”
RW 4192256 SPARSE “u1-s002.vmdk”
RW 4192256 SPARSE “u1-s003.vmdk”
RW 4192256 SPARSE “u1-s004.vmdk”
RW 8192 SPARSE “u1-s005.vmdk”

# The Disk Data Base
#DDB

ddb.virtualHWVersion = “7”
ddb.uuid = “60 00 C2 98 55 91 80 6d-aa c9 e9 78 bd b7 51 f4”
ddb.geometry.cylinders = “1044”
ddb.geometry.heads = “255”
ddb.geometry.sectors = “63”
ddb.adapterType = “lsilogic”
ddb.toolsVersion = “7427”

Written by RS

December 8, 2008 at 10:51 pm

Analysis of Sparse VMDK File

leave a comment »

I did some analysis of the layout of a VMDK file. The graph below shows this analysis: x-axis is the distance in the vmdk file for two logically adjacent blocks(sectors). For for example is block 33 is at offset 3467 and  block 34 is at offset 3468 then the distance in the vmdk file is 3468-3467=1.  The Y-axis is log to the base 2 of the frequency of the distance.

Distance (in VMDK file) of logically adjacent blocks

The graph shows that most logically adjacent blocks have adjacent offsets in the vmdk file. This file had a grain size of 128, so you would expect that only 2/128 sectors would have non-adjacent neighbours (1/64). That is roughly the case. The rest of the distances are clustered around multiples of 128 (as one would expect?). I am not sure if this has to do with NTFS allocating blocks in some sequence or VMW workstation expanding the disk in a particular way.

The vmdk was a windows vmdk 4GB disk with windows XP. Very little use after installing the base OS. Installed a couple of packages like Flex Builder.

Another interesting plot would be to plot the actual distance on the physical disk on the host. Thats next.

Written by RS

December 4, 2008 at 2:46 pm

Compressing Virtual Images

leave a comment »

Experience with running out of storage

Background about sparse files, Windows NTFS and VHD

VHD Pre-Compactor & Compactor

When you create a new VM you can pre-allocate all the storage in one shot or let the storage grow on demand. In the latter case, the VM uses a sparse file

VMDK Wipe & Shrink

3rd Party (Not from MSFT & VMW)

Rsync: for transferring files between machines

Research