Top 7 requirements for infrastructure cloud providers in 2010

This is a summary of the post on the VMOps blog.

1) Inexpensive storage

The storage industry is built on the back of NAS and SAN, but for cloud providers, the overwhelming preference is for inexpensive local disk, or DAS solutions. … every cloud provider I talk with expects storage to be independent of the host physical server, redundant, and provide support for HA.

2) Open source hypervisor

Service providers know that if they plan to compete with Amazon, Rackspace and other cloud providers, on price, VMware is not a good option. Perhaps because it is being used by Amazon, Xen seems to be the most popular hypervisor for Infrastructure clouds among the service providers

3) Integration with Billing and Provisioning Apps

… most hosting companies and MSPs have billing and user management approaches that they have built-up over the years. Every one of the companies I’ve spoken with expect their cloud solution to plug into these existing systems.

4) Image-based pricing to support both Windows and Linux

Most service providers I talk to expect Linux to make up the majority of the images they run int he cloud, but they still need to make sure the cloud will support Windows, and all of the associated technology necessary to manage licenses.

5) Simplicity of administration by end0users

Plenty of end-users will leverage a Clouds API to automatically provision and manage virtual machines, but that doesn’t change the need for a simple UI. Most hosting companies have a huge number of end-users who are used to working with control panels, and an Infrastructure cloud needs to make life easy for these end-users.

6) Reliability

Over the next few years, many of the large providers of dedicated servers will be offering their customers the option to transition to virtual machines running on a computing cloud. For this to be successful, VMs need to offer better reliability than dedicated machines at a lower cost.

7) Turn-key solution

… service providers today can implement a completely integrated cloud stack on commodity hardware, and receive ongoing maintenance and upgrades over the years. Equally important, service providers can license software on a consumption basis, so upfront investment is negligible.

Incidentally, Mr. VMOps Product Manager, you may wish to provide just 3 more requirements to make this a Top 10 requirements list.

Cloud storage predictions for 2010

Detailed post by Sajai Krishnan, CEO, ParaScale is on David Marshall’s VMBlog. The key ideas are summarized below:

The advent of cloud computing has given rise to several cloud storage vendors.

1) The cloud starts to get described

Vendors will begin to describe concrete features and benefits of their product offerings

2) Commodity hardware starts to displace proprietary storage

While all storage vendors claim to use commodity hardware, in reality they are all essentially closed solutions qualified on two or three commodity boxes. Customers are locked into stovepipes with little ability to truly benefit from Moore’s law by selecting from the thousands of commodity servers available at any given point and at multiple points of purchase.

3) Server Virtualization will drive Private Cloud Storage adoption in the Enterprises

With server virtualization, organizations are free to take advantage of low-cost commodity hardware and aren’t tied to proprietary linkage of the OS and the hardware platform. The weak link today is the storage infrastructure behind virtualized servers.

4) A storage middle tier will emerge

The strategic importance of a low-cost, self-managing, petabyte scale tier that provides a platform for analysis and integrated applications emerges in organizations with large stores of file data. These organizations that are investing heavily in new tier1 storage and moving aged data to archive will experiment with a middle tier that leverages low cost commodity hardware and provides read/write access. This middle tier will provide opportunity for administrators to automate storage management and optimize for performance and cost, but at a much lower expense. This middle tier will also support large scale analysis while eliminating related data migration and administrative tasks. The emerging middle tier will also provide an integration layer with service provider cloud offerings. The similar architectures enable "cloud bursting," the seamless ability for service providers to offer spillover capacity and compute to enterprises.

5) Opex, not Capex will emerge as the most important criteria driving storage purchases

Maintenance costs on existing gear will be under heavy review with the emergence of commodity-based hardware storage options.

A year in review: What are our readers looking for?

Our readers are primarily asking questions like:

• How can I free up disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
• Where can I find the best virtual appliances/ Top 10 virtual appliances?
• How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
• Who are the competitors for ec2?

An analysis of the search terms shows interesting clusters:

Serial	Topic	% of queries	Search terms
1	ext4 defragmentation	23%	ext4 defrag, defrag ext4, ext4 defragment, defragment ext4
2	ubuntu ext4 defragmentation	14%	ext4 defrag ubuntu, ext4 ubuntu defrag, ubuntu ext4 defrag, ubuntu defrag ext4, defrag ext4 ubuntu, defrag ubuntu ext4
3	vmware virtual appliance	14%	vmware virtual appliance, vmware virtual appliances, top vmware appliances, top 10 vmware appliances, best vmware appliances
4	virtual appliance	5%	virtual appliance, virtual appliances, top appliances, top 10 appliances, best appliances
5	vmware firewall appliance	5%	vmware firewall appliance, vmware appliance firewall
6	ubuntu defragmentation	4%	defrag ubuntu, ubuntu defrag, defragment ubuntu, ubuntu defragment
7	ec2 competitors	4%	amazon ec2 competitors, ec2 competitors
8	windows 7 virtual appliance	4%	windows 7 virtual appliance, virtual applaince windows 7
9	ext3 defragmentation	4%	ext3 defrag, defrag ext3, ext3 defragment, defragment ext3
10	convert vdi to vhd	3%	convert vdi to vhd, vdi to vhd

If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.

Thank you. I appreciate your interest in this blog.

Compressed VM file transfer using DropBox

I am using DropBox for transferring compressed files including VM’s  between my environment at home, a Mac running Windows XP SP3 in VMware Fusion 2.0.5 and the test machine, a Windows XP SP3 system located in the office lab. Each machines has a DropBox  folder linked to the same account.

Neat product!

I love the simplicity and ease of use. A lot of thought has gone into making the product easy to install, the integration with the host OS (Windows and Mac) is seamless and sets a benchmark for how UI’s for downloadable products should be designed.

Usage model

I compress each file using the Mac’s native file compression and drop into into my DropBox folder. DropBox seems to follow a two-step file transfer process:

1. It first uploads the file completely from the source DropBox folder to the DropBox folder in the cloud
2. After the upload is complete, the file is then downloaded from the DropBox folder in the cloud to the destination DropBox folders.

Setup

Speed ratings are from here. I have been able to correlate these speeds with the end-to-end transfer times.

Transfer Type	Speed Rating for my ISP	Observed DropBox Transfer Rate
Upload	120 KB/sec	70 KB/sec
Download	360 KB/sec	210 KB/sec

Near real-time transfer for uncompressed files

DropBox transfers uncompressed files almost instantaneously between the two machines. The files are transferred sequentially and seem to arrive in order. For example,  I transferred a 1.72 GB folder containing 400 photographs and the photos started appearing sequentially 10 – 15 seconds apart.

Compressed files

Compressed files are transferred as a unit, although dedup applies to blocks contained within it. The transfer times are as recorded below:

Original Size	Compressed Size	Upload Time	Download Time	Total Time
4.30 GB	1.6800 GB	6h 40m	2h 12m	8h 52m
2.15 GB	0.6714 GB	2h 27m	0h 48m	3h 15m
1.10 GB	0.2371 GB	0h 56m	0h 18m	1h 14m

Dedup works well with compressed files

DropBox examines the file to be transferred and builds an index of blocks to be transferred. Its de-duplication technology is smart enough to figure out when not to transfer blocks that are duplicates, i.e., have already been transferred before. For example, when I tried to transfer two clones, the first one took a long time to transfer ( a few hours), but the second transfer was very rapid (under five minutes).

Since I am using the free account, I deleted a 2GB VM from my DropBox folder in order to begin my next transfer. I was pleasantly surprised to see that the next VM transfer was very rapid. I suspect this was because the VM that was transferred earlier was still residing in DropBox’s cache even though I had deleted it, so that DropBox discovered common/duplicate blocks and did not upload them from my Mac.

Summary

Nifty tool. Love it. Will use it a lot.

A few feature requests

• Subfolders: I would like to organize the files by date and category.
• Timers: I would like to time the uploads and downloads easily.
• Profile my usage and suggest how long an end-to-end transfer will take
• Speed up compressed file transfers – improve my effective transfer rate  from ~60% to ~80%- I would like to saturate the available bandwidth for uploads and downloads

Thanks 🙂

Cloud Flavors

1) Software as a Service (SAAS)
Salesforce.com (sales application)
wush.net (svn)

2) Platform as a Service (PAAS)
a) Java
Google App Engine
b) Rails
Joyent
Heroku
EngineYard
c) Python
Google App Engine
http://wiki.python.org/moin/PythonHosting
d) .net
Azure

3) Infrastructure as a Service (IAAS)
Amazon EC2

Amazon EC2 announces developer toolkit for Eclipse IDE

I received the email annoucement from Amazon ec2 earlier today:

We are excited today to introduce the AWS Toolkit for Eclipse, a plug-in for the Eclipse Java IDE that makes it easier to develop, deploy, and debug Java applications on Amazon Web Services. With the AWS Toolkit for Eclipse, you’ll be able to get started faster and be more productive when building AWS applications.

The initial launch of the AWS Toolkit for Eclipse is targeted at Amazon EC2 developers and provides basic management features along with tools for deploying and debugging Java web applications.

The AWS Toolkit for Eclipse, based on the Eclipse Web Tools Platform, guides Java developers through common workflows and automates tool configuration, such as setting up remote debugger connections and managing Tomcat containers. The steps to configure Tomcat servers, run applications on Amazon EC2, and debug the software remotely are now done seamlessly through the Eclipse IDE.

You can read the detailed announcement here and also download the AWS Toolkit for Eclipse.

How credible is EC2’s competition?

Amazon’s Elastic Computing Cloud (ec2) is offering over 1500 AMI’s (virtual appliances). It’s recent foray into Europe highlights its aggressive pursuit of a land grab strategy. So, what has its competition in the US been up to? Every major web hosting vendor has announced a cloud initiative:

• The Rackspace cloud is called Mosso. Its prices compare favorably with Amazon’s. Its pricing calculator and spreadsheets can be found here.
• Savvis has a VMWare ESX based offering for Windows and Linux
• AT&T is also offering VMWare ESX based hosting Windows, Linux and Solaris x86. They intend to offer Citrix XenServer and Microsoft Hyper-V in the future. AT&T offers SLA’s for cloud services, a first.
• Terremark has also launced an Enterprise Cloud, which incidentally is a utility computing cloud sans virtualization
• IBM’s cloud initiative has its products available as AMI’s on ec2.
• Sungard has not yet publicly announced its cloud plans.
• Verizon is slated to announce its cloud initiative in Summer 2009

Amazon’s ec2 is nearly three years old now (Mar 2006) and competition has started appearing on the horizon just over the past year. If you look at the timescale of announcements,

• Rackspace’s Mosso is just over a year old (Feb 2008),
• Terremark’s cloud is nine months old (June 2008),
• AT&T’s cloud is seven months old (Aug 2008),
• IBM’s AMI launch on ec2 and Savvis’ cloud is three weeks old (Feb 2009).

Since ec2 was first to market, the Rackspace, AT&T and Savvis cloud offerings have a “me too” feel to them. However, unlike other vendors, Rackspace has published pricing on the Web and it appears to be very competitive with ec2. 

Unlike Amazon, Savvis and AT&T are going the buy versus build route to get fast time to market. They are initially launching their service using VMWare technology put together using Professional Services instead of following Amazon’s approach of building a proprietary infrastructure using Open Source software as its foundation. In fact, this may be the preferred route amongst the upper echelons of cloud service providers. There is optimism that providing cloud services is a growth business. I am noticing startups like Enomaly (“Build your own private elastic cloud”) and VMOps (“launch ec2 today”, aka public cloud) offering cloud infrastrcture products.

In conclusion, credible competition is emerging and there are real alternatives to ec2 available today. However, given the state of the economy, I think the market will begin to form by 2010 and should reach critical mass by 2011-2012.