Part one of this article, Cloud Computing and Open Source, presented an overview of several cloud computing solutions available today that are friendly to open source. In this article we'll focus on one of these solutions, Amazon's Elastic Compute Cloud (EC2), and walk through the development of an open source mashup for EC2. Amazon EC2 is a top player in the cloud computing space and gives organizations the ability to leverage world-class compute resources on a pay-as-you-go basis. It also provides a rich set of APIs based on open standards and the ability to mix and match other Amazon services such as Simple Queue Service (SQS), Simple Storage Service (S3), SimpleDB, and Elastic Block Store (EBS), to build innovative applications for the cloud.

Setting Up Your EC2 Environment

Two concepts at the core of EC2 are an Amazon Machine Image (AMI) and an instance. An AMI is an encrypted file comprising the operating system, application, data, configuration files, and dependencies required to run your application. Once an AMI is launched, the running system is referred to as an instance. To manage your EC2 resources, well-documented web service interfaces (e.g., SOAP and Query APIs) are provided for use by your programs or from the command line.

The SOAP or Query interfaces are supported in major programming languages such as C++, C#, Java, Perl, Python, PHP, and Ruby. AWS also provides a set of Java-based command line tools (EC2 API Tools) that can be used in shell scripting environments.

To use EC2, you need an Amazon Web Services (AWS) account. Once your AWS account has been set up, you then need to sign up for EC2. During the EC2 signup process, an Amazon S3 account is created for you. This S3 account is required by EC2, since the AMI used to launch an instance is stored on S3. Each AWS account is assigned an Access Key Id and a Secret Access Key that must be used to secure and authenticate all API calls to EC2. The Access Key ID and Secret Access Key can be found under the AWS Access Identifiers section of your Amazon Web Services account page.

To use the EC2 API command line tools, you need to create a X.509 certificate and RSA key using the same AWS Access Identifiers page. Now you're ready to setup and install the EC2 API command line tools on your local Linux system. Create a directory named .ec2 in your home directory ($HOME).

>  mkdir $HOME/.ec2

Next, download the X.509 certificate you just created by clicking on the Download X.509 Certificate button and save it in $HOME/.ec2/cert-<STRING>.pem. Download the private key by clicking on the Download Private Key File button and save it in $HOME/.ec2/pk-<STRING>.pem.

Now, define two environment variables, EC2_PRIVATE_KEY and EC2_CERT, which are required by the command line tools to authenticate requests to EC2. EC2_PRIVATE_KEY points to the file path of the private key ($HOME/.ec2/pk-<STRING>.pem). EC2_CERT points to the file path of the X.509 certificate ($HOME/.ec2/cert-<STRING>.pem).

> export EC2_PRIVATE_KEY=$HOME/.ec2/pk-<STRING>.pem
> export EC2_CERT=$HOME/.ec2/cert-<STRING>.pem

Next, download the EC2 API Tools from the Amazon EC2 Resource Center. Unzipping the version of the zip file downloaded at the time this article was written will create a directory named ec2-api-tools-1.3-24159. Define an environment variable EC2_HOME with the full path to the directory containing your tools version (e.g., $HOME/ec2-api-tools-1.3-24159 in our case). The command line tools depend on this environment variable to locate supporting libraries.

> export EC2_HOME=$HOME/ec2-api-tools-1.3-24159

Change 1.3.24159 to the specific version of the api tools you have downloaded. For your convenience, you may want to add $EC2_HOME/bin to your system $PATH variable.

> export PATH=$PATH:$EC2_HOME/bin

The last step of the command line tools setup is to install a Java environment. A JRE or JDK (version 5 or later) environment works. On our Ubuntu 8.04.1 desktop system, here are the steps we followed to install a Java runtime environment.

First, verify that the 'multiverse' repository is enabled in /etc/apt/sources.list, you should see a line similar to

deb http://us.archive.ubuntu.com/ubuntu/ hardy multiverse

Execute sudo aptitude update to update the list of available packages from the apt sources. Now, install the Sun Java 6 JDK package by running sudo aptitude install sun-java6-jre. Verify that the installation was successful by running java -version.

Define the JAVA_HOME environment variable pointing to the directory that contains a sub-directory named bin, which in turn contains the java executable. The command line tools require this environment variable to locate the Java runtime executables.

> export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre

The command-line tools are now configured to "talk" to EC2.

Setting Up and Launching Your Instance

We'll now step through the process to setup and launch an EC2 instance.

EC2 provides great flexibility when it comes to choosing an AMI for your instance. You can either build a custom AMI from scratch, modify an existing AMI, or select an existing one from the list of Public AMIs at the Amazon Web Services Resource Center. Images for various Linux distributions such as RHEL, CentOS, Fedora, Ubuntu, and Debian are available.

EC2 has a concept of public and private AMIs. Public AMIs can be launched as instances by any registered EC2 user, enabling them to remix, reuse and share. Private AMIs can be used by the publisher of the AMI and any EC2 users they have granted permissions to.

For our sample mashup application, we selected an existing public AMI with a base installation of Ubuntu Hardy (8.04.1). Each AMI has an AMI ID associated with it. This ID uniquely identifies the machine image within EC2.

Secure access to the instance requires creation and subsequent use of a public/private keypair. The public key is stored in the instance and the private key is stored locally on our system. To generate the keypair, we ran the following command. Note that the first argument ec2_ubuntu_keypair is a name for the keypair.

> ec2-add-keypair ec2_ubuntu_keypair

Copy and paste the output of this command into a text file called id_rsa_ec2_ubuntu_keypair. Change the file mode of this file to user-only-read/write by running chmod 600 id_rsa_ec2_ubuntu_keypair.

Now we're ready to launch an instance by running the ec2-run-instances command below. Note that we've specified the AMI ID (ami-179e7a7e) and the keypair name (ec2_ubuntu_keypair) as arguments.

> ec2-run-instances ami-179e7a7e -k ec2_ubuntu_keypair
RESERVATION     r-3b0bdf52      <AWS_ACCOUNT_ID>    default
INSTANCE        i-0e0bd167      ami-179e7a7e    pending ec2_ubuntu_keypair       0  m1.small 2008-08-25T06:02:50+0000       us-east-1a      aki-a71cf9ce    ari-a51cf9cc

It takes a few moments for the instance to boot up. You can check on the boot progress of the instance with the ec2-describe-instances command.

> ec2-describe-instances
RESERVATION     r-3b0bdf52      <AWS_ACCOUNT_ID>    default
INSTANCE        i-0e0bd167      ami-179e7a7e    ec2-75-101-199-88.compute-1.amazonaws.com       ip-10-251-109-189.ec2.internal      running ec2_ubuntu_keypair       0               m1.small        2008-08-25T06:02:50+0000    us-east-1a      aki-a71cf9ce    ari-a51cf9cc

The output from ec2-describe-instances command indicates that our instance has booted up and is running. By default, network access to the instance is turned off. We ran the commands below to allow traffic on port 22 (ssh), 80 (http), and 443 (https). Other ports can be opened up in the same way.

> ec2-authorize default -p 22
GROUP           default
PERMISSION              default ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0

> ec2-authorize default -p 80
GROUP           default
PERMISSION              default ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0

> ec2-authorize default -p 443
GROUP           default
PERMISSION              default ALLOWS  tcp     443     443     FROM    CIDR    0.0.0.0/0

To login to the instance, we used the following ssh command.

> ssh -i id_rsa_ec2_ubuntu_keypair root@ec2-75-101-199-88.compute-1.amazonaws.com

When you're done using your instance, remember to shut it down. To shut down an instance, run the ec2-terminate-instances specifying the instance ID (i-0e0bd167) as the first argument. You can also shutdown an instance by logging on to it with ssh and running the shutdown -h now command.

> ec2-terminate-instances i-0e0bd167
INSTANCE i-0e0bd167 running shutting-down

Building the Mashup Application

One of the most compelling features of EC2 is its rich set of APIs to manage every aspect of this complex and dynamic infrastructure. These APIs make EC2 an attractive platform for open source developers. Let's look at how we might build a mashup to extend the power of EC2. In our mashup we wanted to demonstrate how to integrate the functionality of these APIs to collect management information about the EC2 instance we just created. And we wanted to use ubiquitous browser-based open technologies such as Javascript and AJAX.

A great example of an open source browser tool to manage EC2 resources is Elasticfox. Elasticfox, originally developed by James Greenfield at Amazon, is a popular Firefox extension that comes with an Apache 2.0 license. It combines Javascript, AJAX and Mozilla's XUL to communicate with EC2 using the SOAP and Query APIs. Elasticfox also provides an excellent starting point for building your own Javascript interface to EC2 and S3. So we decided to use some of the Javascript libraries included with Elasticfox in our mashup.

What We Did

Our mashup assembled a subset of the Javascript libraries used by Elasticfox and customized a handful of Javascript interface functions. Together, these custom functions request and then display detailed information about our EC2 instances and the AMI's available to our account. The information sent back by EC2 is displayed in a simple HTML table.

Since we're using a pure Javascript interface, we had to provide a proxy for the AJAX HTTP request from our non-Amazon HTML page to Amazon's EC2. Because Elasticfox uses XUL and is integrated into Firefox, it can avoid the browser's normal cross-domain scripting constraints. We cannot. An excellent personal proxy solution can be put together using Tinyproxy with the help of a small configuration generator script that we have provided with our mashup. A second proxy solution, which is more scalable, involves using mod_rewrite and mod_proxy plugins for the Apache Web Server. We have provided instructions and a configuration generator script for this approach as well. If you're using Linux, setting up Apache as a proxy is a snap on your local box. We've included sample Apache mod_rewrite rules to provide the required proxy service to Amazon.

A Closer Look at the Code

The code we have used to illustrate the Javascript interface to EC2 has the following structure.

<HTML> Page
  <HEAD> Section
    Elasticfox Javascript library link--aws.js
    Our custom Javascript functions--describeInstances, onCompleteDescribeInstances (in ec2_instances.html)
                                       describeImages, onCompleteDescribeImages (in ec2_images.html)
  <BODY>
    Text input fields to enter your AWS Access Key and AWS Secret Key
    <DIV>--populate with dynamically built table

There are two HTML pages, one for gathering information about our EC2 instances (ec2_instances.html) and another for querying the AMIs available to our EC2 account (ec2_images.html).

Let's examine the EC2 instances information page, ec2_instances.html. The highlights of this page are the "custom" Javascript functions found in the <HEAD> section. These are actually dynamic redefinitions of functions used in Elasticfox Javascript libraries by the XUL framework. The first function, describeInstances, is a modification of the same function in Elasticfox's ec2ui_controller object defined in controller.js. This has been modified to provide a wrapper for the lower-level client facilities. The wrapper sets up the required EC2 credentials and invokes the client query action to "describeInstances". When EC2 replies with an XML response, the modified callback function ec2ui_controller.onCompleteDescribeInstances is invoked. This function decodes the XML response and builds the HTML table that is dynamically inserted into the DOM and displayed on the Web page.

You can do much more with the Amazon APIs. For example, next steps might involve modifying and building your own version of Elasticfox. It's open source and has build and package assembly scripts that make it easy to work with as well as to integrate with your own projects. Elasticfox is hosted on SourceForge. Another place to explore is the AWS Developer Connection web site. Its EC2 Community Code section provides a wealth of open source code examples, documentation, and tutorials.

Issues We Faced

A big impediment for any mashup using AJAX is the browser's "same-origin policy." Normally a browser, for very good security reasons, prevents access to any site other than where the Web page originated. This sand-box delivers a lethal blow when trying to automate Web services using Javascript and AJAX because you cannot simply combine services like EC2 with data from Google or Yahoo or even information from your own local services. The most common solution to this problem is to provide an external server that coordinates and proxies connections to services across different remote locations.

Fortunately, there are some great open source packages that can make complex AJAX mashups feasible on both a small, personal level as well as on the largest enterprise scale. These packages can be used to build a specially configured proxy to hide from the browser the actual target resources of the mashup service you are assembling. Our AWS example highlights both the personal level solution, using the latest version of Tinyproxy, as well as illustrates a larger scale solution using the dynamic rewrite machinery of Apache.

Conclusion

This article covers the combining of open source technologies to build a programmatic interface to cloud computing. First we explained how to set up an EC2 environment and then we developed a simple application using AJAX to illustrate how to mashup services from different resources (in our case local resources together with Amazon EC2). Along the way, we have offered two solutions for the cross-domain scripting problem that often cripples the effective use of AJAX as a mashup technology. You can use these techniques to extend your web application across a variety of cloud services. You can remix useful information from services like Flickr, Facebook, Google, Amazon and present them all together on a single page.

Try out this mashup by downloading the project. For reference, take a look at the AWS Mashup release notes below.

AWS Mashup Release Notes

The files for aws_mashup, contained in the archive aws_mashup.tar.gz, are organized by function into Javascript libraries, several shell scripts, HTML pages, README setup guides, and binary executables of Tinyproxy built for 32-bit and 64-bit Linux. They are:

• aws.js: Elasticfox Javascript library (selected packages)--We have made small modifications to this library to provide for cross platform XMLHTTP and XPATH interfaces. It has been tested on Firefox 2&3, Opera 9, IE6, Safari 3.
• javeline_xpath_konq.js: Javeline XPATH Library--We have modified Javeline's pure Javascript XPATH library for our project to run in Konqueror 3&4. Konqueror is the only browser we've tested that lacks native XPATH support.
• configure_project_for_apache.sh: Shell script to configure aws_mashup project for Apache2
• configure_project_for_tinyproxy.sh: Shell script to configure aws_mashup project for tinyproxy
• ec2_images.html: HTML page containing query interface to EC2 for describing images (AMIs)
• ec2_instances.html: HTML page containing query interface to EC2 for describing instances
• setup_java_for_hardy_live_cd.sh: Shell script to setup required Java environment on Hardy Live CD
• README.SETUP_FOR_APACHE: List of steps to setup environment for Apache2
• README.SETUP_FOR_TINYPROXY: List of steps to setup environment for tinyproxy
• run_tinyproxy.sh: Shell script to run tinyproxy
• tinyproxy: ELF 32-bit executable
• tinyproxy_64: ELF 64-bit executable