Hyperbio » Cluster http://hyperbio.net Startups and running Tue, 17 Jan 2012 12:06:31 +0000 en hourly 1 http://wordpress.org/?v=3.3 eBay’s enormous data warehouses http://hyperbio.net/2009/04/30/ebays-enormous-data-warehouses/ http://hyperbio.net/2009/04/30/ebays-enormous-data-warehouses/#comments Thu, 30 Apr 2009 14:39:08 +0000 Leila Boujnane http://hyperbio.net/?p=970 Curt Monash meets with Ebay’s Oliver Ratzesberger and gets us numbers on two of the world’s largest data warehouses in the world. Look at these Ebay stats!

Metrics on eBay’s main Teradata data warehouse include:

  • >2 petabytes of user data
  • 10s of 1000s of users
  • Millions of queries per day
  • 72 nodes
  • >140 GB/sec of I/O, or 2 GB/node/sec, or maybe that’s a peak when the workload is scan-heavy
  • 100s of production databases being fed in

Metrics on eBay’s Greenplum data warehouse (or, if you like, data mart) include:

  • 6 1/2 petabytes of user data
  • 17 trillion records
  • 150 billion new records/day, which seems to suggest an ingest rate well over 50 terabytes/day
  • 96 nodes
  • 200 MB/node/sec of I/O (that’s the order of magnitude difference that triggered my post on disk drives)
  • 4.5 petabytes of storage
  • 70% compression
  • A small number of concurrent users

More details.

]]> http://hyperbio.net/2009/04/30/ebays-enormous-data-warehouses/feed/ 0