Over the past 5 years, systems and applications software development teams have progressively adopted virtual machine platforms from VMware, Microsoft and Xen to provide self-contained environments for developing and testing their products. The primary drivers for this trend are:
- Server consolidation: Data center real estate and electrical power are serious resource constraints that are driving companies to consolidate computing to fewer, high capacity servers on which tens to hundreds of discrete virtual machines are instantiated and run. This trend impacts the software development community as well, where developers in development teams share a large server with others in their team and achieve isolation for their code through the use of VM’s.
- Distributed development teams: Large software companies such as Adobe, CISCO, Check Point Security, HP, IBM, Microsoft, Oracle, Synopsys, etc., have development organizations based in the US, China, Eastern Europe, Israel and India. Smaller organizations, too, either have in-house or subcontracted development teams based in the US and, China or India.
Sharing VM’s between teams
VM images are usually shared between teams developing and testing loosely-coupled components that communicate with each other over a public or a proprietary API, or standalone components that can be invoked remotely, The loose coupling permits each team to develop and test their component independently and provide their counterparts a stable version of their component for integration and testing in one or more VM images.
Limitations for sharing code through the source code control repository
Traditionally, distributed development teams in large organizations have operated successfully by setting up a distributed source code repository that is remotely synced periodically over the network, e.g. IBM’s Rational ClearCase, SVN plus rsync, etc. Each team uses a common build procedure for compiling and linking the binary executable for incrementally integrating new code and testing. This model works well for a monolithic product or a component of a product but does not support the development team effectively when:
- There are loosely coupled components that are rely on standardized API’s for communication, e.g., TCP/IP, JDBC/ODBC, HTTP, SOAP and do not require synchronization of development tasks through the source code control repository.
- The component is supported across several different Windows OS versions and Linux OS/filesystem versions, so the development and testing must span all the supported platform without having to dedicate physical hardware for each one of them
Limitations of using a shared file systems
Development teams that are on the same LAN and can access a shared filesystem over the network, use network mounted shares for sharing VM’s. However, this sort of sharing is not feasible even within the same company when the shared filesystem is not accessible from other subnets within the corporate network. For geographically distributed teams, the network latency involved with accessing such a shared filesystem from a remote location often makes its use impractical.
High latency for VM image transfers
Transferring a multi-GB file over the LAN, WAN or Internet is a royal pain! To get a first-hand feel for how long it takes, sign up for an account with SkyDrive, 50GB’s of storage provided by Microsoft in the cloud, and try to upload a 5 -10 GB file over the Internet. You’ll see that It takes very long. The same kind of latency is experienced when files are transferred over the internal LAN; the latency is acute when the teams are distributed across continents and the VM’s are transferred over the Internet. It is often very expensive to get a reliable high band with Internet connection in foreign countries and the transfers have to be initiated after business hours to ensure they complete overnight.
Limitations of WAN acceleration
Large companies that maintain private networks can use WAN acceleration for reducing the latency for large file transfers. However, this is not feasible for most companies that don’t have a substantial investment in their own network infrastructure. WAN acceleration is effective only within the private corporate network and does not help when large VM files are to be transferred to an external contractor performing QA on the product, or when customers want to send a VM that encapsulates the environment within which a product failure was observed.
Increased disk space and budget consumption
Developers often maintain several virtual machines on their personal workstations/laptops, each running a version of the software that reflects a particular stage of the development life cycle. Each VM image is a very large file ranging from a few GB to 10’s of GB in size and consumes a lot of disk space on their personal workstations/laptops.
While developers routinely create new VM’s, or snapshots of stable VM’s, during the development life cycle, older and obsolete VM’s seem to rarely get deleted. When the disk on their personal workstation/laptop gets full, they migrate the older VM’s to a shared drive on the network – this could be a Windows file share in small-scale to medium-scale companies, NAS devices like Network Appliance filers in medium-scale to large-scale organizations, or a SAN in large-scale Enterprises.
The proliferation of VM’s on personal workstations/laptops and on shared storage has a budgetary impact – the Line Manager who owns the budget for this project has to either purchase larger internal disks as replacements, or augment with additional external storage, or pay IT through budget dollars or chargebacks for shared storage that is consumed.