Today I was very fortunate to have gotten a tour of Facebook’s new datacenter up in Prineville, Oregon (map). This datacenter is the most energy efficient in the world and only a handful of press got a look. We’ll have a video up after editing it, but here’s a look at the datacenter in photos. I shot all of these photos on an unmodified iPhone 4 with Instagram, that just got an update today. For the panoramic photos I was using Occipital’s 360 app.
Here’s the sight that we saw on arriving. Keep in mind this building is HUGE and there’s a sizable solar array out front (here’s a panoramic photo from inside that solar array), which doesn’t really power much of the datacenter, but powers some of the buildings around the site. Photos don’t really do it justice, but think about three average Walmarts put end-to-end :
Facebook is so big that it has its own flag:
Walking in, yes, we are in the right place:
Just past the Facebook sign is a monitor in the lobby that shows you the state of the datacenter and how well the cooling systems are working:
Inside the security door the local community made these quilts, which is their interpretation of what a social network looks like:
Walking in Thomas Furlong, director of site operations at Facebook, brought us into a huge series of rooms which “process” the air. First room filters the air. Second room filters it further.
Here’s Thomas showing us one of the huge walls of filters (these filters are similar to the ones in my home heating system, except here Facebook has a wall of them).
Here’s a better shot of just how massive this filtering room is:
Then the air goes into a third room, one where the air is mixed to control humidity and temperature (if it’s cold outside, as it was today, they bring some heat up from inside the datacenter and mix it here) and on the other side, there’s a huge array of fans, each of which has a five horsepower motor (today the fans were moving at 1/3 speed, which makes them more efficient).
Here you can see the back sides of one of the huge banks of filters:
Here Thomas stands in front of the fans:
Here’s a closeup look at one of the fans that forces air through the datacenter and through the filtering/processing rooms:
Finally, the air moves through one final step before going downstairs into the datacenter. In this final step small jets spray micro-packets of water into the air. As the water evaporates, which it does very rapidly, it cools the air. One room I didn’t take photos in was filled with pumps and reverse osmosis filters, which makes the water super pure so it works better when using it to cool in this way. One final set of filters makes sure no water gets into the datacenter. Here’s a closer look at the array of water jets:
Here you can see the scale of the room that sprays that water:
Here’s a closeup of one of the jets of cooling water:
Finally we got to follow the air down into the datacenter where there was a huge floor with dozens of rows. Each row had rack after rack of servers.
Here Thomas stands in front of just one of those racks:
This 180-degree view gives you a look down the main corridor (on the side you can see is only half the datacenter — these are the newer “Open Compute” servers, the other half they asked us not to take pictures of, and that contained their older server technology).
If you click here you can see a panoramic photo of one of these rows.
What does this all mean? Well, for one, it brings jobs to Prineville, which is a small town with about 10,000 residents in a very rural county (we drove about half an hour through mostly farmland just to get to Prineville). But listen to Prineville’s mayor to hear what it means for her community.
Which brought up the question: why Prineville. The execs who showed me around today said they chose the site based on an exhaustive search for the perfect combination of low-seismic risks, cooler and mostly dry weather, access to power and Internet trunk lines (Prineville is an old railroad community, and fiber lines run under the railroads here) and a variety of other factors including low tax rates and friendly climate to business, etc.
Anyway, it’s not often that you get to see inside a modern datacenter. You’ll be reading more about this tour, since there were other journalists there as well, hope you enjoyed these early pictures.
By the way, why did Rackspace send me there? For those who don’t know, I’m a full-time employee of Rackspace which is the world’s biggest web hosting company.
Because we’re already building a datacenter based on the “Open Compute” plans that Facebook made and put into Open Source (the datacenter as well as the specs for the machines is all in open source now). More on Open Compute here. Plus we’re datacenter geeks so love seeing how other companies do it so we can learn from what they’ve done.
Amazing!!
a litlle scaring) looks odd)
I intended to be cynical and poopoo this post when I clicked the link, but this really is f**ing cool.
I intended to be cynical and poopoo this post when I clicked the link, but this really is f**ing cool.
I intended to be cynical and poopoo this post when I clicked the link, but this really is f**ing cool.
It was even cooler being there in real life. The thing is massive. Sure shows that Facebook is a real company and not just a fly-by-night startup!
Fly-by-night? No. Agreed. These guys are the new walmart of the internet. Anything you want all on one site. Games, friends, search, email and good Oregon wine inbound shortly.
Yeah, because after 6 years in business, and subsequently worth billions of dollars, a company might be misconstrued as a fly-by-night company.
It was even cooler being there in real life. The thing is massive. Sure shows that Facebook is a real company and not just a fly-by-night startup!
It was even cooler being there in real life. The thing is massive. Sure shows that Facebook is a real company and not just a fly-by-night startup!
It was even cooler being there in real life. The thing is massive. Sure shows that Facebook is a real company and not just a fly-by-night startup!
Great post Robert. Thanks for sharing.
Just to think this was built as a result of one man’s passion… and they say the economy is failing… I think that old business stopped innovating, new business is finding a way to make it work!
Correction, this was all built due to millions of peoples obsessions.
Of course! They also have everything stored on other datacenters as well and have contingency plans in case the entire datacenter goes down.
What do we make of this statement? “According to a press release issued by Greenpeace, Facebook uses “about 55% coal power while Google uses 34% and Yahoo uses just 12.7%.””
I’d say that Facebook is normal. 55% is roughly how much electricity coal supplies in the United States. Those numbers are entirely dependent on where the datacenters are located, as well. Facebook just happens to place their datacenters in places where coal is predominant source of power. Big deal.
Not really a big deal, though Coal is a HORRIBLE source of electric power. Enormously destructive on terrain and water/air.
Clean coal energy, what a total 100% contradiction in terms.
maybe nuclear instead??
or the others perhaps.. solar/wind/geo-thermal?
You know, what’s commonly used for renewable energy resources… nuclear isn’t renewable.
Nuclear is the ultimate renewable, it just requires a longer time horizon.
I’m in complete agreement, the problem is, coal is still so cheap nothing can compete. Nuclear is the next closest, but with its instability still ongoing in Japan I think we should seriously fund ‘green’ alternatives solar and wind. But with the coal and oil lobbies I don’t it getting enough government help – and how can private industry compete without that? The hole is too big.
There is no coal in Oregon. But Oregon has a plus. We have the Bonneville Dam and a wind farm which produces all of the electricity for Oregon. What ever electricity FB with use will come from what Oregon produces.
We make of this statement that Greenpeace is a slightly crazy or irrational organization at times, since that’s just a reflection of what the power grid looks like where they are, I presume, mitigated *slightly* by the amount of solar panels the data centers have built.
Congrats to Mark Zuckerberg..
That’s what I was thinking. Quite a leap from his small web app he built on his laptop.
Whoww! Thanks Robert. I am a lover of the datacenter and infrastructure:)
@Scobleizer How do you say this?
“This datacenter is the most energy efficient in the world ” Any measures out there?
Execs from Google and Microsoft were touring today and they didn’t argue with the claim. Also, Facebook made that claim over and over a week ago and no one reputed it. So, it stands.
No one reputed it? Did anyone refudiate it?
(Nice photos. Thanks for sharing them.)
I think unless you prove otherwise the claim would stand. Not that actually attempting to prove/disprove the claim is a bad thing.
Google:
“The trailing twelve-month (TTM), energy-weighted average PUE for all of these facilities is 1.16, exceeding the EPA’s 2011 goal for state-of-the-art data center efficiency.”
http://www.google.com/corporate/datacenter/efficiency-measurements.html
Facebook claims 1.15…then uses an 8 hour period measurement to come up with 1.07…
The Prineville facility is expected to have a Power Usage Effectiveness (PUE) rating of 1.15.
http://www.datacenterknowledge.com/the-facebook-data-center-faq-page-3/
The 1.07(with is accompanying notes below)
PUE calculated at full load over an 8 hour period during the commissioning stage in December 2010. We expect our PUE to fluctuate over time and will report it on a quarterly basis.
http://opencompute.org/
Thanks for the spin guys…it would be nice for them to wait to actually find an annual or even quarterly average, but who cares about that – they can claim 1.07 over 8 hours!!!
the fb data center is DEF FAP worthy. HOLY SH!T BALLZ Batman. *_*
lol “low-seesmic risks” awesome =)
In hindsight that IS funny!
Nice photos, thanks for making the trip to share these with us Robert. Thanks to FB as well for putting their design where others will benefit.
One of the big takeaways from this is, no matter how much we talk about “the cloud”, every bit and byte is still on a physical machine(s) somewhere. Server hardware must surely be the next top hardware “thing”.
F**king huge! larger than any football field in Indonesia. That’s why I never get an answer “SERVFAIL” when I dig facebook.com
Yup, and keep in mind that Facebook has several datacenters around the world and more on the way.
Wondering what % of their total operation is running out of this new facility ?
Wondering what % of their total operation is running out of this new facility ?
I am glad you love Panaroma so much.
Yeah, they are a little harder to do than a normal photo, but are so good in places like these. It’s really difficult to share the scale of a building like this. Hopefully a bit of that came through.
is there a tool that maps a place in 3d using geolocation and then sort of photosynthes it together from all the images people take from the tour? maybe the true magic behind color could do that?
http://www.photosynth.net/ may be what you’re looking for.
is there a doom/3d shooter map of the datacenter? i’m sure it’d be the hell of a facebook game : hide’n'seek in our datacenters
how does the energy efficiency and overall design compare with that of the other big player like microsoft and google who are always said to have really good and efficient datacenter designs? (the ms azure deployment system e.g. seemed really clever 2 years ago)
It’s hard to know because Microsoft and Google don’t share their latest numbers.
This is an article 2yrs ago on Google’s datacenters http://news.cnet.com/8301-1001_3-10209580-92.html
No 2011 numbers up yet, but up to date data through all of last year.
http://www.google.com/corporate/datacenter/efficiency-measurements.html
(sorry for mass-asking questions, they just keep coming to my head)…
is there a crowdsourcing system for servers? i mean: how much energy, traffic and compute capability is just wasted all over the world by private computers idling in the net (just updating tweets or pseudo processing emails)? with a clever system a fraction of each could be used to create a highly redundant server network – kind of like bittorrent meets seti .. data security should be easy by cleverly distributing pieces (microtask-like in a way)…
Any details on storage media being used ( i.e. solid state vs. hard drive )?
Each server has a Seagate hard drive in it. I’m not sure if there are some computers that have lots of SSD in them. I’ll try to find out. Look at OpenCompute, though, and they share their specs for the computers.
Each server can have 384gb of memory: http://opencompute.org/specs/Open_Compute_Project_AMD_Motherboard_v1.0.pdf So, I bet some of these things are loaded with SSD.
This is really cool. It seems that there is a lot of opportunity for innovation in the world of data centers. This is great .. thanks for posting!
This is really cool. It seems that there is a lot of opportunity for innovation in the world of data centers. This is great .. thanks for posting!
Awesome! I wish I could go to work for them!
I’m blown away by the massive amount of data that now passes through Oregon. Hats off to Mayor Roppe and the people of Prineville for landing a big one!
scary what’s needed to run FB u’d think all that processing power could be used to do something far more wothwhile.
600 million monthly active users and growing seem to thing it’s pretty worthwhile :-p
That may not be the best metric ?
OK, howabout over 300 million active users every day?
Cool
Thanks for the report
Thanks for the write up and the great photos. It is also good to know Rackspace is on board with open datacenter designs. Thanks.
This is a really great post, Robert, for several reasons:?1) level of detail 2) ability to capture the spirit of Facebook, which is trying hard NOT to be just a corporation (quilts, site selection, solar). I knew this from the HQ tour Randi gave me. 3)demonstration of what it takes to run an online site with 600m visitors demanding that it never be down. You captured all that. Thanks.
Wow, that just looks really cool! Can you imagine? Wow.
http://www.web-anonymity.at.tc
Some really great shots. Thanks for the sneak peek. Looking forward to the video next.
All in all, it looks a lot like the data I use to run my blog – http://www.groovypost.com
Ok fine… My DC might be just a “tad” bit smaller.
Cool! It looks just like the HELIOS One array in FallOut New Vegas!
That’s one hell of a lot of servers and engineering for one hell of a lot of pokes / sheep / werewolves / embarrassing photos / mafia crimelords / complicated relationships / procrastinators
You just summed up the entire internet
The State of Oregon put in miles and miles of fiber in the sparsely populated area east of the Cascades many years ago. They caught hell because 99.9% was dark – no users, no prospects. I was one of those who shook my head at the “waste” of money. No more. With Google and Facebook data centers now in place the investment looks better. Planning a data center? Come to Oregon!
Doesn’t it look a bit like a highly advanced computer jail?
Why not set up the datacenter somewhere in north Canada? could have avoided so much cooling!
Taxes probably. Plus you need cheap power and a fat Internet pipe….
Sent from my Mobile
That’s an interesting thought. I wonder if, during some sort of “slow” period (does fb have a slow period?), they could donate their cpu’s to genome research, cancer research, etc.
Slow period…. Probably not considering the world never sleeps.
Sent from my Mobile
Nitpick: You have Seesmic on the brain. “combination of low-seesmic risks”
WOW. Super cool pics and I didn’t really expect anything less from the social giant!! Thanks for sharing this Robert and I must admit I wish I was there to see it.
Well done
Holy crap…. that looks so huge.
That’s what she said…
HAHAHAHHAHAHHA
some clever engineering going on over there, but Prineville? sounds like zyngas next big thing..
Very nice write-up, Robert. I envy you the opportunity to walk the facility! Aside from the fantastic job they did on the facility design, their new servers are very impressive, too… 38% more efficient, while 24% less costly than anything else on the market. You can see some numbers on the facility and the new servers on this post: http://docsheldon.com/i'm-no-facebook-fan-boy-but…/
The fact that they’ve put all their drawings and specifications for both the servers and the facility up for public access is to be applauded, too.
I work for a major server vendor and we could never get away with building systems like this. Frankly I don’t know how they do either, but I guess it is because they don’t sell them. These systems would never pass any FCC classification. They have no grounding or electrical shielding, so they would emit massive amounts of electrical noise.
There isn’t much around there for at least 30 miles, so I doubt the electrical noise is a big deal.
By the way, both my Verizon and AT&T iPhones worked fine in the datacenter (which is funny because they don’t work in downtown San Francisco), so electrical noise sure isn’t a big deal to those. If it exists at all. And I doubt you’re right about not grounding them. How can you figure that out from the photos?
awesome Facebook data center! is it possible for common people to have the chance to tour
Incredible. Great work and thank you!
Dang it Scoble! You made me buy a new app. (360 Panorama)
If you don’t like it in a capitalist society vote with your dollar. Turn off your internet connection.
facebook != the internet
it’s not a direct feed to them… geez
That’s like saying if you don’t agree with Walmart, don’t drive.
they’re there for one reason: cheap hydroelectric power on the Columbia River.
Because my DSLR is in the shop. The audio board broke on it.
I know they’re all capable video cameras now, but there’s something very wrong with that statement.