Yahoo Search Architecture Based on Hadoop

How to implement internet scale search based on open-source technologies?

Yahoo has solved this problem using the open-source Apache Hadoop distributed computing framework. Hadoop provides a distributed filesystem (HDFS) and support for the MapReduce distributed computing metaphor that is also the foundation of Google's architecture.

The Yahoo! Search Webmap Hadoop application runs on a more than 2,000 node Linux cluster each with two quad core processors and produces data that is now used in every Yahoo! Web search query.

The Webmap build starts with every Web page crawled by Yahoo! and produces a database of all known Web pages and sites on the internet and a vast array of data about every page and site. This derived data feeds the Machine Learned Ranking algorithms at the heart of Yahoo! Search.

Some Webmap size data:

• Number of links between pages in the index: roughly 1 trillion links
• Size of output: over 300 TB, compressed!
• Number of cores used to run a single Map-Reduce job: over 10,000
• Raw disk used in the production cluster: over 5 Petabytes

Smaller companies who cannot afford to run their own Hadoop cluster can combine it with the Amazon EC2 ans S3 cloud computing service. The result? Open source utility computing on Google scale!

Amazon EC2 Flash Video Streaming

The ability to host web video clips in the Cloud is getting near with the release of the Wowza Pro media server for Amazon EC2 (beta).

Wowza Media Server Pro Unlimited is a high-performance, extensible and a fully interactive, Flash media server for live and on-demand flv streaming, chat and recording.

Wowza Media Systems has teamed with Amazon Web Services to make billing quick, easy, and secure for the combination of Wowza Media Server Pro, machine time with various EC2 instances and bandwidth.

Pricing of Wowza Pro Unlimited for Amazon EC2 is:

• Instances
• $0.14 per Small Instance-hour consumed (or part of an hour consumed)
• $0.45 per Large Instance-hour consumed (or part of an hour consumed)
• $0.88 per Extra-Large instance-hour consumed (or part of an hour consumed)
• Data Transfer
• $0.12 per GB of data transfer in
• $0.20 per GB of data transfer out
• Monthly Charge
• $5.00 recurring monthly charge

This pricing includes all charges for the use of Wowza Pro instances on Amazon EC2 (both Wowza Pro licensing and Amazon EC2 fees).

It is still in beta so improvements might be expected related to

• performance
• load-balancing
• using Amazon S3 for hosting videos
• management

Streaming video using Amazon EC2 and S3 is a great utility computing and storage application. It could help startups with limited resources to compete with youtube and other video hosting services. Check out the user guide and the forum to learn more.

Recommended books on Flash Video Streaming:

• Hands-On Guide to Flash Video: Web Video and Flash Media Server
• Flash Video for Professionals: Expert Techniques for Integrating Video on the Web
• Networking IP Video: Video on your IP Network - Streaming, Surveillance, Conferencing, Telepresence, and Video Telephony - and How to Deploy them all (Cisco Technology Series)
• Hands-On Guide to Video Blogging and Podcasting: Emerging Media Tools for Business Communication

It is Time to Learn Amazon EC2 and S3

Utility computing is here to conquer the IT world! Amazon Vice President Adam Selipsky has recently unveiled the newest statistics on usage of Amazon S3. Their 265,000+ developer community has now stored over 10 billion (10,000,000,000) objects in S3 at a rate of 27,601 transactions per second. That's huge progress compared to the 5 billion objects just a few months ago!

Programmers and web developers should really look forward to books such as Programming Web Services - S3, EC2, SQS, and FPS by James Murty. The book will cover the most disruptive and useful web services available today:

• Amazon Simple Storage Service (just launched in Europe!)
  
• Amazon Elastic Compute Cloud (now in beta with new instances)
• Amazon Simple Queue Service (offers reliable and scalable hosted queue)
• Amazon Flexible Payments Service (still in limited beta)

To illustrate the power of the Amazon Web Services check out this blog entry by Derek Gottfrid: Self-service, Prorated Super Computing Fun!

The New York Times has decided to make all the public domain articles from 1851-1922 available free of charge. These articles are all in the form of images scanned from the original paper. In fact from 1851-1980, all 11 million articles are available as images in PDF format. To generate a PDF version of the article takes quite a bit of work!

Derek has achieved this with the help of Amazon S3/EC2 and Hadoop!

I quickly got to work copying 4TB of data to S3. Next I started writing code to pull all the parts that make up an article out of S3, generate a PDF from them and store the PDF back in S3. This was easy enough using the JetS3t — Open Source Java toolkit for S3, iText PDF Library and installing the Java Advanced Image Extension.

For deployment, I created a custom AMI (Amazon Machine Image) for EC2 that was based on a Xen image from my desktop machine. I logged in, started Hadoop and submitted a test job to generate a couple thousands articles — and to my surprise it just worked. It churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3.

Now that this adventure can be called a success, I can’t imagine how we might have done it without Amazon S3 / EC2 . The one caveat I will offer to people who are interested in doing something like this is that it is highly addictive.

Indeed an interesting utility computing success story!

Erlang: The Programming Language for Multicore CPUs

The future of computing is going to be concurrent. Sun has just announced the UltraSPARC T2 CPU which they call the World's fastest microprocessor. With 8 cores and 64 threads it makes a true system on a chip. Even desktop systems have multi-core processors nowadays. However traditional software is not well prepared to effectively utilize large number of cores.

Concurrent programming is hard. Most programming languages do not make it easier either. On the other hand Erlang is ideally positioned for this new world. It was designed from the ground up to take advantage of parallel and multi-core architectures.

Erlang is a concurrent functional programming language and runtime system. It was designed to support distributed, fault-tolerant, soft-real-time, non-stop applications. Erlang was originally a proprietary language within Ericsson, but was released as open source in 1998.

Erlang programs usually scale very well on multi-core systems. Joe Armstrong explains why:

"Back in the old days (20 odd years ago) there were two models of concurrency:

• Shared state concurrency
• Message passing concurrency

Now the whole world went one way (towards shared state), and we went the other"

The Erlang concurrency model differs from other languages by not having any shared state. If a process wants to communicate with another, it does so by sending messages. This method scales better than methods that uses shared memory for communication.

You can learn much more about Erlang in Joe Armstrong's new book:
Programming Erlang: Software for a Concurrent World

It's very readable and does not require prior experience with functional languages. The book is packed with examples and encourages experimenting; in fact the first chapter explains the installation of Erlang. A reviewer calls it "the most important programming language book this decade".

To have some fun check out this article and video from 1990: Erlang Now!


More resources:

• Erlang for the practical man - tutorial on using RSS, POP3, SSL and HTTP in Erlang
• ErlyBird - Erlang IDE based on NetBeans
  

YouTube Architecture Unveiled

YouTube has grown rapidly to serve 100+ million videos per day. How could they manage this incredible growth? What is the architecture behind YouTube that supports this extreme scalability?

In this Google Tech Talk Cuong Do discusses the scalability challenges that have arisen during YouTube's short but extraordinary history. Cuong is currently an engineering manager at YouTube/Google. He was part of the engineering team that scaled the YouTube software and hardware infrastructure from its infancy to its current scale.

Interesting bits:

• Initial team consisted of 2 sysadmins, 2 architechts, 2 developers, 2 network engineers and a DBA
• Based on Apache, Python and MySQL
• Most popular content is moved to a CDN (content delivery network)
• Much more details in the video and notes...
  

The notes of the Tech Talk are also available on the High Scalability Blog.