Showing posts with label utility computing. Show all posts
Showing posts with label utility computing. Show all posts

Friday, July 25, 2008

IT in 2018: From Turing’s Machine to the Computing Cloud


How will the IT landscape look like in 2018? This is the question asked by Nicholas Carr, the author of "The Big Switch: Rewiring the World, from Edison to Google". He takes readers of this free Internet.com eBook on a trip one decade into the future to see how IT as we know it will change and what those changes mean for IT professionals, hardware and software vendors, and corporations of all sizes.

This interesting journey starts at 1936 in Cambridge where Alan Turing has invented the idea of the universal computing machine.

"With enough memory and enough speed, Turing’s work implies, a single computer could be programmed, with software code, to do all the work that is today done by all the other physical computers in the world.

And that is why the modern corporate data center, with all its complex and expensive stacks of machinery, is on the path to obsolescence."

Carr clearly explains the current hot virtualization and consolidation trend:

"Turing’s discovery that “software can always be substituted for hardware” lies at the heart of “virtualization,” which is the technology underpinning the great consolidation wave now reshaping big-company IT. As the cost of computing power and storage capacity has continued its decades-long freefall, it’s become possible to turn more and more hardware into software code – to use a single powerful computer to run many virtual machines.

All the pieces of hardware stuffed into corporate data centers – not just servers but storage drives, load balancers, firewalls, switches, and even the cables connecting the gear – exist, after all, to carry out instructions. Virtualization simply turns the hardwired instructions into code and gets rid of the physical machinery. That not only saves a lot of cash, it makes the radical automation of formerly manual IT processes possible. Once IT infrastructure turns into software, it can be programmed, easily and from afar. Code, as always, replaces labor. "

Check out his eBook to take the journey to the conclusion:

"Just as the last century’s electric utilities spurred the development of thousands of new consumer appliances and services, so the new computing utilities will shake up many markets and open myriad opportunities for innovation. Harnessing the power of the computing grid may be the great enterprise of the twenty-first century."

Monday, February 4, 2008

Amazon EC2 Flash Video Streaming

The ability to host web video clips in the Cloud is getting near with the release of the Wowza Pro media server for Amazon EC2 (beta).

Wowza Media Server Pro Unlimited is a high-performance, extensible and a fully interactive, Flash media server for live and on-demand flv streaming, chat and recording.

Wowza Media Systems has teamed with Amazon Web Services to make billing quick, easy, and secure for the combination of Wowza Media Server Pro, machine time with various EC2 instances and bandwidth.

Pricing of Wowza Pro Unlimited for Amazon EC2 is:
  • Instances
    • $0.14 per Small Instance-hour consumed (or part of an hour consumed)
    • $0.45 per Large Instance-hour consumed (or part of an hour consumed)
    • $0.88 per Extra-Large instance-hour consumed (or part of an hour consumed)
  • Data Transfer
    • $0.12 per GB of data transfer in
    • $0.20 per GB of data transfer out
  • Monthly Charge
    • $5.00 recurring monthly charge
This pricing includes all charges for the use of Wowza Pro instances on Amazon EC2 (both Wowza Pro licensing and Amazon EC2 fees).

It is still in beta so improvements might be expected related to
  • performance
  • load-balancing
  • using Amazon S3 for hosting videos
  • management
Streaming video using Amazon EC2 and S3 is a great utility computing and storage application. It could help startups with limited resources to compete with youtube and other video hosting services. Check out the user guide and the forum to learn more.

Recommended books on Flash Video Streaming:

Thursday, January 31, 2008

Amazon EC2 and S3 Surpassed Amazon.com


Amazon has announced in its fourth quarter earnings release that the bandwidth usage of the Amazon Elastic Compute Cloud (EC2) and Simple Storage Service (S3) has surpassed all of Amazon.com’s global website traffic combined. This milestone means that the 330,000 registered Amazon Web Services (AWS) developers collectively use more bandwidth than the web retailer itself. Amazon has become the leading cloud computing service provider with its own internet platform.

The Amazon Web Services solution catalog lists more than a hundred solutions based on AWS including innovative backup, archival, community and media hosting services by web 2.0 startups and developers.

The resources of AWS Developer Connection grows rapidly. Check out these featured tutorials and articles:
The Amazon Cloud propels new innovation by changing the economics of scalable computing and storage resources. Let's prepare for the Big Switch!

Monday, January 14, 2008

The Big Switch: Rewiring the World, from Edison to Google

I have found this amazon.com review today:

Don't walk, run to your nearest bookstore to read Carr's dazzling THE BIG SWITCH. If you can only read one book in 2008 this should be it. The writing is clean, pure as spring water and thoughtful. Nicholas Carr makes the coming "information utility" simple for the layman to understand. The description of the development of electricity and its impact on society is fascinating and lays the groundwork for the likely outcome of the information age over the next few decades. I enjoyed this book which is a must for anyone interesting in the future of the information utility.

I am looking forward to receiving my copy of The Big Switch: Rewiring the World, from Edison to Google.

The Big Switch's full ramifications are still far from clear, but the areas it has touched so far are profoundly changed. We are further along in The Big Switch than most people suspect, and the momentum to continue is unstoppable.

The Big Switch represents a great opportunity for the next wave of internet innovations. In fact utility computing or cloud computing is maturing quickly thanks to Amazon EC2, Google and many others. Let's get ready for a great ride!

Friday, December 14, 2007

Amazon SimpleDB - Scalable Cloud Database

Amazon has announced the limited beta of Amazon SimpleDB - a simple web services interface to create and store multiple data sets, query your data easily, and return the results. Together with the Simple Storage Service (S3), Elastic Compute Cloud (EC2) and other web services Amazon offers a complete utility computing platform. SimpleDB was the missing piece of AWS - the scalable structured database.

Amazon SimpleDB is easy to use and provides the core functionality of a database - real-time lookup and simple querying of structured data like:
  • CREATE a new domain to house your unique set of structured data.
  • GET, PUT or DELETE items with attribute-value pairs
  • QUERY your data set using this simple set of operators
Users pay only for the resources that they consume. Pricing is based on
  • Machine Utilization
    $0.14 per Amazon SimpleDB Machine Hour consumed
    (normalized to the hourly capacity of a circa 2007 1.7 GHz Xeon processor)
  • Data Transfer
    $0.10 per GB - all data transfer in
    $0.13-$0.18 per GB - data transfer out
  • Structured Data Storage
    $1.50 per GB-month
Get started today with the Amazon SimpleDB Guide.

Many companies already use Amazon Web Services as their utility computing infrastructure like finalists of the Amazon Web Services Start-Up Challenge:
  • Ooyala - a platform to deliver video content for publishers
  • Brainscape - an engine for measuring the networks in the brain
  • Commerce360 - optimize paid search campaigns to deliver improved campaign efficiency
  • Justin.tv - a massively scalable live video platform
  • MileMeter - “auto insurance buy the mile”
  • UserTesting.com - quick and cheap website usability testing
  • WeoGeo - a one-stop marketplace for mapping
With the introduction of Amazon SimpleDB utility computing just got even better!

Tuesday, November 6, 2007

It is Time to Learn Amazon EC2 and S3

Utility computing is here to conquer the IT world! Amazon Vice President Adam Selipsky has recently unveiled the newest statistics on usage of Amazon S3. Their 265,000+ developer community has now stored over 10 billion (10,000,000,000) objects in S3 at a rate of 27,601 transactions per second. That's huge progress compared to the 5 billion objects just a few months ago!

Programmers and web developers should really look forward to books such as Programming Web Services - S3, EC2, SQS, and FPS by James Murty. The book will cover the most disruptive and useful web services available today:
  • Amazon Simple Storage Service (just launched in Europe!)
  • Amazon Elastic Compute Cloud (now in beta with new instances)
  • Amazon Simple Queue Service (offers reliable and scalable hosted queue)
  • Amazon Flexible Payments Service (still in limited beta)
To illustrate the power of the Amazon Web Services check out this blog entry by Derek Gottfrid: Self-service, Prorated Super Computing Fun!

The New York Times has decided to make all the public domain articles from 1851-1922 available free of charge. These articles are all in the form of images scanned from the original paper. In fact from 1851-1980, all 11 million articles are available as images in PDF format. To generate a PDF version of the article takes quite a bit of work!

Derek has achieved this with the help of Amazon S3/EC2 and Hadoop!

I quickly got to work copying 4TB of data to S3. Next I started writing code to pull all the parts that make up an article out of S3, generate a PDF from them and store the PDF back in S3. This was easy enough using the JetS3t — Open Source Java toolkit for S3, iText PDF Library and installing the Java Advanced Image Extension.

For deployment, I created a custom AMI (Amazon Machine Image) for EC2 that was based on a Xen image from my desktop machine. I logged in, started Hadoop and submitted a test job to generate a couple thousands articles — and to my surprise it just worked. It churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3.

Now that this adventure can be called a success, I can’t imagine how we might have done it without Amazon S3 / EC2 . The one caveat I will offer to people who are interested in doing something like this is that it is highly addictive.

Indeed an interesting utility computing success story!

Thursday, September 20, 2007

Internet Platforms Defined

Platforms on the Internet are one of the hottest topics these days. Marc Andreessen helps to overcome the current confusion of the platform concept in his incredibly insightful blog post. Marc has tackled the concept of internet platforms and defined the three levels of them. Here is a short summary but I recommend you to check out the full article for details.

A "platform" is a system that can be programmed and therefore customized by outside developers -- users -- and in that way, adapted to countless needs and niches that the platform's original developers could not have possibly contemplated, much less had time to accommodate.
  • A Level 1 platform's apps run elsewhere, and call into the platform via a web services API to draw on data and services -- this is how Flickr does it.
  • A Level 2 platform's apps run elsewhere, but inject functionality into the platform via a plug-in API -- this is how Facebook does it. Most likely, a Level 2 platform's apps also call into the platform via a web services API to draw on data and services.
  • A Level 3 platform's apps run inside the platform itself -- the platform provides the "runtime environment" within which the app's code runs.
The most interesting of these internet platforms are those that are level 3. Some examples of these are:
  • Akamai's EdgeComputing
  • Amazon S3, EC2 and FPS
  • Ning (Marc's own company)
  • Google Apps could became a level 3 platform
  • Salesforce.com
  • Second Life
The platforms of the future will be in the cloud as online services that you will tap into over the Internet. This is a fundamental change that will drive a lot of innovations. We are living in very exciting times!

Wednesday, September 5, 2007

How To Deliver High Definition Video on the Internet

Today more and more households have HDTV sets and broadband internet connections with 6-8+ MBit/sec bandwidth. However the scalable delivery of HD video content is still a big challenge.

The capacity to deliver high definition video already exists at the edge of the network – leveraging the massive build-outs of individual networks. The challenge comes from how you tap into that capacity. The best way to solve this problem is to deliver from the edge using distributed architectures.

The potential benefits are huge and the demand for high definition video is growing. How to deliver HD content to the internet? Let's examine the possibilities.

1. Use Content Delivery Networks (CDNs)

Content Delivery Networks such as Akamai have tons of servers distributed across hundreds of cities. These servers can deliver high bandwidth content such as HD video from the Edge to the users. Even YouTube moves most popular content to a CDN. Akamai has recently announced their plans to bring High Definition to the internet. Their platform is architected to meet the following technical requirements:
  • Technology and an operational model to operate serving devices in the largest high-throughput networks around the world
  • Established relationships with the largest high throughput networks
  • Support for delivery, storage, and management of files greater than 2 Gigabytes
  • Support of VC-1 and MPEG-4 video standards
  • Support for files with resolutions of 720p, 1080i and 1080p
2. Use Utility Computing services such as the Amazon S3 and EC2

Utility computing and storage services are another way to host scalable video content and applications. The Amazon Simple Storage Service (S3) provides such an on demand distributed service. However these services are not yet proven for hosting High Definition content. Check out my previous post on Amazon S3 for Video Hosting.

3. Use Peer-to-Peer (P2P) protocols such as BitTorrent

P2P file sharing applications have been popular in the last couple of years. P2P could be used for TV and movies as well:
  • Joost offers a new way of watching TV using P2P as the underlying architecture
  • Vuze offers Hi-def Theater using their BitTorrent based client software
Although P2P could be an efficient way to distribute content it can have significant bandwidth overhead for the users.

Which way to go?

The optimal choice depends on the requirements. CDNs are proven technologies but P2P could be cheaper as it matures. Utility storage services are similar to CDNs and the two could became competitors. What is your choice?

Tuesday, August 28, 2007

Amazon S3 for Video Hosting

In an earlier post I have written that the next YouTube could be built on Amazon’s Web Services (AWS) by two guys in a dorm for roughly $0. It is indeed very easy to host videos on Amazon S3.

All you need to do is to sign up for an Amazon S3 account and upload your videos with the Amazon S3 Firefox Organizer (available here: S3Fox). To see this in action check out this interesting Amazon S3 video hosting tutorial by Perry Lawrence on YouTube.

You can simply include the hosted files in your web pages. However it is also possible to create nice web applications or mashups using the video content on S3. The book Amazon.com Mashups by Francis Shanahan will help developers to get familiar with Amazon Web Services. The examples in this book demonstrates how to integrate AWS with APIs from Yahoo!, eBay, Google and YouTube. Check out this sample chapter on the Amazon Simple Storage Service (S3) available at Wrox.

It has never been easier to create your own YouTube!

Monday, August 6, 2007

Amazon EC2, S3 + Hadoop = Open Source Utility Computing on Google Scale

Google is the undisputed King of Scalability. The High Scalability blog collected the open secrets of the Google Architecture. Two important components are:
These distributed technologies are the foundations of their scalable and reliable storage and processing clusters.

The open source Hadoop project implements these functions so you can build a Google like cluster in your data center. In case you do not have 100s of servers at your disposal ask Amazon for help. The Amazon EC2 and S3 utility computing services can host your cluster at a reasonable price. Check out this tutorial by Tom White who illustrates how to use Hadoop and Amazon Web Services together using a large collection of web access logs:
Yahoo has recently announced their support for Hadoop.

"Looking ahead and thinking about how the economics of large scale computing continue to improve, it's not hard to imagine a time when Hadoop and Hadoop-powered infrastructure is as common as the LAMP (Linux, Apache, MySQL, Perl/PHP/Python) stack that helped to powered the previous growth of the Web."

Monday, July 30, 2007

Low Cost Online Backup with Amazon S3

How do you backup your critical data? Traditional backup procedures use regular copy sessions to tape, disk or CD-R/DVD-R media. The broadband internet allows companies to provide online backup services. Such online systems can easily scale and provide cost effective remote backup for a wide range of customers. However they are definitely not one-size-fits-all.

Consumer oriented online backup offering such as Carbonite, Mozy or XDrive has a typical price of ~$1/GB/month although limited free services are available. Check out this review on ars technica for details.

Another solution is to use the Amazon Simple Storage Service (S3) with the s3sync tool or the Amazon S3 Firefox Orgainzer plug-in (S3Fox). Pricing ($0.15/month for a GB of stored data) is really attractive compared to the above services.

Find out more in these practical how to guides:

Wednesday, April 18, 2007

Amazon + Two Guys + $0 = Next YouTube

Utility computing and storage is changing the way web services are built.

The Amazon Simple Storage Service (Amazon S3) is storage for the Internet. It is designed to make web-scale computing easy and affordable:
  • $0.15 per GB-Month of storage used.
  • $0.20 per GB of data transferred (cheap bandwidth makes this possible)
The next YouTube will be built on Amazon’s Web Services by two guys in a dorm for roughly $0 (see SmugBlog). Amazon S3 now has 5 billion objects stored in it. What's more, it had 920 million S3 requests on its peak day, and 16,607 requests in its peak second (see ReadWriteWeb).

Sun Microsystems offers Utility Computing @ network.com for $1/CPU-hr.

The Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easy and affordabe at $0.10/instance-hour! It is in limited beta but already popular.

Google has many assets and APIs related to utility computing. All kind of data can be stored in Google Base. Their web-scale distributed computing platform is well known. This paper on the Google Filesystem gives us a peek into the future of Web applications and platforms.

Who will build the next YouTube on these utility computing platforms?