Open Synapse

Hi, I'm Abhay Kumar. I like to throw up interesting things I encounter here. You can also see the result of my mustache pact with my coworkers, check out what I'm up to or see the few photos I take. I like to receive email and instant messages (AIM / Jabber), as well. Some links of interest: Calais Text Tagger, Powerset

Feb 26
Permalink
» World's Largest Hadoop cluster

Yahoo switched over to their new cluster which is freaking huge. Here are some stats:

Some Webmap size data:

  • Number of links between pages in the index: roughly 1 trillion links
  • Size of output: over 300 TB, compressed!
  • Number of cores used to run a single Map-Reduce job: over 10,000
  • Raw disk used in the production cluster: over 5 Petabytes

Damn! (via O’Reilly Radar)

Comments (View)
blog comments powered by Disqus

Change CongressVote Obama