I started using EC2 back in 2008 and just recently joined the Private Preview of Google Compute Engine (GCE). Currently, I use the cloud to build ad-hoc MPI clusters for simulating protein-protein interactions using a software package called OSPREY (Open Source ProteinREdesign for You). I have been using EC2 and MIT's brilliant StarCluster to spin up clusters with up to 800 virtual cores and am now looking at GCE as a possible alternative. This series of blog posts will detail my experiences and observations and possibly useful insights.
Before I begin, a few things. First, I am highly biased; I love both Amazon (long time Prime member) and Google (just look at my gmail address). Second, a number of blogs have offered detailed looks at GCE (here, here, and, here among others) and I would highly recommend reading the overview here. This post looks at price (as of 1/1/2013) and instance positioning.
Pricing and Instance Characteristics
Instance pricing drops not infrequently and current pricing across all instance types can be found here for EC2 and here for GCE. Please note that this comparison is for what Amazon calls On-Demand Instances. Amazon offers many ways to access instances including Reserved Instances, where you can prepay for light, medium and heavy utilization of instances over longer periods of time, and Spot Instances, where you can bid lower prices for underutilized infrastructure.
Instance types compete on numerous characteristics including:
- processing capability
- ephemeral storage (size and speed)
- IO performance
and a number of others (boot speed, networking configuration, performance consistency, etc) that I won't get into here.
Apples to Apples or Apples to Oranges?
Let's start with processing capability. Unfortunately, both Amazon and Google don't make it web-simple to compare the processing power of their respective virtual instances. So let's dig deeper. EC2 measures processing power with "EC2 Compute Units" which are, per Amazon:
"One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation."
which is really less than helpful. However, Amazon released the "Cluster Compute Eight Extra Large Instance" on 11/14/2011. This monster offers 88 EC2 Compute Units but adds in that this is really equivalent to "2 x Intel Xeon E5-2670, eight-core "Sandy Bridge" architecture." Intel says that its E5-2670 is a 2.6GHz, 32nm core with 8 actual cores capable of running 16 hyper-threads (2 threads per core). Since the CCEELI has 2x8 cores = 16 cores = 32 hyper-threads we have the following relationship:
88 EC2 comp = 32 Hyper-threads (@2.6 GHz, Sandy Bridge) or 2.75 EC2 Compute Units = 1 "Virtual" Core (@2.6GHz, Sandy Bridge)
Now, let's look at Google's "cuter" measurement for processing power, the "GQ."
"GCEU (Google Compute Engine Unit), or GQ for short, is a unit of CPU capacity that we use to describe the compute power of our instance types. We chose 2.75 GQ’s to represent the minimum power of one logical core (a hardware hyper-thread) on our Sandy Bridge platform."
Reading further, for the current generation of 'n1' machine types, "a virtual CPU is implemented as a single hyperthread on a 2.6GHz Intel Sandy Bridge Xeon processor." Does this sound familar? Thus,
2.75 GCEU = 1 Hyperthread (@2.6GHz, Sandy Bridge)
Piecing it all together we get:
1 GCEU = 1 EC2 Compute Units
which, assuming the two platforms are running identical processors, is quite the coincidence. Let's run with this for the moment.
Assuming Some Things Equal
What everyone seems to care about is the distribution of CPU vs RAM, ephemeral storage is just that; your instance goes down and everything in ephemeral storage is gone. It should be faster but it can always be bolstered with cloud storage either Amazon Simple Storage Service (S3) or Google Cloud Storage.
Thus, we have three key variables: price, CPU, and RAM. Google and Amazon instance categorizations reaffirm this as both offer "standard", "high memory", and "high CPU" instance types.
In the scatter plot of compute power and instance memory, one thing becomes obvious immediately. Most instance types cluster in the lower left of the plot, having 25 GQ's and 60 GB of RAM or less. However, Amazon (circles) sells access to a few outlier instances for more niche types of computing needs.
The outliers in the plot above (listed below) are all from Amazon, which has seen and attempted to fill market opportunities.
- Tan, circle, top middle = Cluster Compute Eight Extra Large Instance (60.5 GiB memory/88 EC2 Compute Units/3370 GB of instance storage)
- Purple circle, far right = High Storage Instances (117 GiB memory/35 EC2 Compute Units/24 hard disk drives each with 2 TB of instance storage)
- Light green circle, middle = High I/O Quadruple Extra Large Instance (60.5 GiB memory/35 EC2 Compute Units/2 SSD-based volumes each with 1024 GB of instance storage)
Excluded from this plot is Amazon's Cluster GPU Quadruple Extra Large Instance (22 GiB of memory/33.5 EC2 Compute Units/2 x NVIDIA Tesla “Fermi” M2050 GPUs/1690 GB of instance storage) as I have no good way of factoring in the general purpose GPU computing power (and neither does Amazon, or so it seems.)
When we eliminate Amazon's outlier instances listed above, we see a very interesting story. Google instances (triangles) are positioned to offer greater compute performance in most cases.
To counter attack, Amazon released its second generation "M3" instances on 10/31/12, offering the same amount of memory but more computational muscle (red/orange line). The M3 instances stand out due to offering 3.25 EC2 Compute Units per virtual core, suggesting that the underlying hardware is faster than the Xeon E5-2670 Sandy Bridge @ 2.6GHz.
Interestingly, neither provider offers a single instance that gives you access to an entire virtualized processor. Such an instance serving up the entire 8 core Xeon E5-2670 would clock in at 44 compute units.
Looking at the ratio of CPU to RAM across instance classes from both Google and Amazon, we see some strong consistency:
- Standard Instances range from 0.58 to 0.87
- High Memory Instances range from .38 to .42
- High CPU Instances range from 2.9 to 3.1
What About Price?
If we compare the M1 Medium in EC2 to the n1-standard-1-d, you get the same 3.75 GB of RAM but 37.5% more compute units for only 6% increased cost. This identical trend continues comparing the M1 Large to the n1-standard-2-d, and the M1 Extra Large to the n1-standard-4-d. I would not be surprised to see another price drop on first generation Amazon instance types.
Second Generation Instances
Things get interesting when we compare the second generation Amazon M3 images to GCE's. Comparing the M3 Extra Large to n1-standard-4, we see the EC2 instance offers 18% more compute at 20% more cost. This exact pattern continues examining the M3 Double Extra Large to the n1-standard-8
Comparing the High-CPU Medium to the n1-highcpu-2-d, the GCE instance offers 5.8% more memory, 10% more compute for 3% more cost. The gap closes slightly for High-CPU Extra Large versus the n1-highcpu-8-d, which has 2.8% more memory, 10% more compute units, for 3% more money.
Competition and What's Next
An arm's race and/or price war in the cloud computing space will almost certainly benefit the end consumer. Per some reports, Google is targeting GCE not at the little guy or websites but at very large scale enterprise operations requiring many instances. It will be interesting to see how Google competes with the very rich ecosystem built atop Amazon's service, one that will take some time to replicate. In my next article, I plan to benchmark comparable instances to see if the assumptions made about the hardware were correct.
Some early benchmarks were posted mid-2012 but I suspect that they may no longer be valid.
A Note on Naming
Amazon's instance naming scheme is getting out of hand, just look at the "Cluster Compute Eight Extra Large Instance." If this doesn't stop, selecting an instance type from Amazon is going to sound a lot like ordering at Starbucks with your free drink card .