Valleywag on “drunken employee” in datacenter

http://www.canix.ca/a0_news.php?langue=en

BeerCo has an exclusive deal with a large cage owner at the Canix datacenter in downtown Montreal. We can set you up with Gigabit and or 10/100 Megabit connections at low enterprise prices for data reciprocation, and duplication purposes.

We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.

We can set up a light duplication system at Canix so that only a light version of your services run while your other center gets back up and working. So you do not have to cache all the data in duplication. We can do this at a very small enteprise class cost, and know the people at the center well.

We will set up all the software and hardware for you, for Linux OR windows, so that you only need to set up a duplication client on your side to shuttle the data in secure SSL. We will make a custom and highly scalable solution for you, no matter the time it takes.

Can you afford not to?

I deliver.
Thanks everybody.

Oh care went into it, they have journalists. But pot calling kettle black, you blog whatever that play-dough brain of yours happens to oatmeal out.

And to pre-answer the question, NO, you obviously can’t run your own nameservers. I suggest UltraDNS which we use for our SN site, as we can make a CURL based php application to go in an automate changes to the DNS for an instant and un-noticable switch to Montreal from San Fran.
No one will realize what happened, and we keep all customers 1000% confidential.

And to pre-answer the question, NO, you obviously can’t run your own nameservers. I suggest UltraDNS which we use for our SN site, as we can make a CURL based php application to go in an automate changes to the DNS for an instant and un-noticable switch to Montreal from San Fran.
No one will realize what happened, and we keep all customers 1000% confidential.

During regular uptime BeerCo pumps beer through their Gigabit pipes:-)

Robert,

I’ve worked in MANY data centers in my career. If an employee with full access (never give full access to anyone) to everything wanted to really throw the cat in among the pigeons, it wouldn’t be hard.

Here are a few things that could be done if someone was to go rogue with too much leverage.

- On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises. Kill VRRP. All the IP space could be advertised so the router(s) are targets for bots and worms, etc. Blackhole/null routes could be put in place for entire IP ranges. Routers should be watched and the logs looked at everyday by qualified people.

- Firewall rules could be altered/established to allow bad ports and services pointed at mission-critical machines.

- TTL could be set on the DNS server entries to the maximum setting so dynamic websites do not update when content is updated. Push changes out to the root servers.

In a data center, jobs SHOULD be compartmentalized. Firewall guys work firewalls, router guys work routers, DNS guys do DNS. It’s been this way everywhere I’ve ever worked. No one ever had the keys to the Lamborghini, so the speak. Good security is more than just firewalls, it’s a process, procedure, it’s limited access.

Worse, most data centers allow their people to have SSH access to servers, routers, and firewalls. Kerberos, SSH, and RSA keyfobs offer no security against inside threats. Believe it or not, quite a few threats happen from WITHIN an organization. From its own people.

Trust but verify. EVERYTHING.

They knew it was BS when they printed it:
“We’re sure 365 Main will deny that such a thing could ever happen. And, conveniently, the neighborhood is having power troubles, too.”

The absurdity of that last line is phenomenal…”Yes, the suspect was conveniently dead at the time of the crime, but how else can you explain…”

My daughter is in law school and Valleywag has come up in class discussion. They’ll probably be sued out of existence one day when they cross the line and the wrong party.

“- On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”

What are routers?

Unix right. Take the /etc/shadow and toss it through the dictionary. With a powerful enough cipher throughput, it shouldn’t take very long.
No miscreant is going to sit there and bother with a 32 char password. If you know what you’re doing, nobody is going to be able to seize your infrastructure.

“During regular uptime BeerCo pumps beer through their Gigabit pipes:-)”

If your company pays, we’ll find a solution to whatever IT issue you may throw at us, and we’ll do it with a smile. Beer through gigabit pipes included. We’ll say yes.

“- On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”

What are routers?

Unix right. Take the /etc/shadow and toss it through the dictionary. With a powerful enough cipher throughput, it shouldn’t take very long.
No miscreant is going to sit there and bother with a 32 char password. If you know what you’re doing, nobody is going to be able to seize your infrastructure.

“During regular uptime BeerCo pumps beer through their Gigabit pipes:-)”

If your company pays, we’ll find a solution to whatever IT issue you may throw at us, and we’ll do it with a smile. Beer through gigabit pipes included. We’ll say yes.

Or if the router is ROM boot only and you don’t have a prompt, you can simply reset it, and restore the backup of the settings you professionally and regularly backed up to secure media and put in a safe place.

Chris,

Infrastructure doesn’t need to be seized. A dedicated outsider or even insider could use a bot network and DDoS your routers into oblivion.

Long passwords are rather useless in any event. I’m not advocating using names or dictionary words. I use nothing but passphrases on all my machines.

For example, you could take a lyric from a song and turn it into a passphrase. Just make sure the passphrase isn’t an actual word.

Mary had a little lamb, it’s fleece was white as snow.

Take the first letter from every word…

Mhallifwwas. Could be made better…

Mh@1lifWwaS

Can get better with a little thinking…

The thing to do on servers is to create several root passwords and give each responsible admin a separate one. This way you know who did what.

SSH, Kerberos, RSA keyfobs. etc., all need to be used. Together. Two or three-factor authentication. What you know (ssh passwd), what you know (kerb passwd), what you have.

Set kerberos’ time clock skew to 1 minute. Make sure employess’ laptops for remote access are synchronized with the kerberos time server clock. Make it a thinking man’s game. The more employees have to think, the less mistakes they make.

One more VC: I’m a journalist, by training, too. That doesn’t mean that what you read here is journalism.

Whenever I’ve been the subject of Valleywag stories I’ve found they NEVER call for my reaction, or to give my side of the story. They only reprint stuff that’s emailed into them from readers. Most of the time it turns out to be false because they can’t tell whether they are being hoaxed or not. They just like to be entertaining. That they certainly are, but don’t say it’s “journalism.” They even admit it’s a gossip magazine. I guess you think the National Enquirer is “journalism” too.

Sigh.

Whether you’re a journalist, a blogger or a gossip monger, you can still be sued for libel. Of course, Robert knows this, but I’m surprised by the number of people who think it’s only “journalists” that are beholden to defamation laws and that “freedom of speech” means you can publish anything you want to without consequence.

“But it’s ridiculous to say that someone could “mess up” a rack”

Not really. “A rack” generally refers to the physical rack itself as well as all of the equipment housed on it. Slace through the neatly organized twisted pair cables, damage the rack fans, and a few other things I can think of doing with a screw driver, would pretty much “mess up a rack” for a day or two at least.

I’m not so sure about the big red button. That would be disruptive for sure, but most modern hardware, particularly of the server variety, is designed to survive having the plug yanked.

The Valleywag story is obviously wrong, but interestingly, PG&E still says the cause is unknown.

@13 “I’m a journalist, by training, too”

Why? Because you took some classes at SJSU? I’m pretty sure they want you to stop admitting that.

“I guess you think the National Enquirer is “journalism” too.”

Uh..by your past definitions, it is,

>>The absurdity of that last line is phenomenal…

Dawn, it’s the absurdity that ensures that Valleywag will NOT be sued out of existence. They know what they’re doing.

Talk to your daughter again after she graduates from law school and practices First Amendment law a few years… then she’ll know what she’s talking about. Law students don’t know much of anything, and I say that from direct experience.

Not defending Valleyway Robert, just saying. Old lawyers die hard.

You’re wrong Scoble. I spend all day in data centers (multiple ones, well known big ones and smaller ones too) and an employee with the right set of keys could wreak havoc in a matter of minutes. You could walk up, unlock racks and start jerking out cables (power, network). Each rack would takes less than 30 seconds to completely hose for a while (not total destruction, just chaos). Network cables would be worthless, power downs, etc. Give me 5 minutes with a master key (oh yes they exist) and I could ruin the weeks for several admins.

Brian, I know that libel cases are next to impossible to win, but there are people out there, people like Donald Trump as an extreme example, who will sue them for the sheer principle of it.

The National Enquirer has been sued a lot and there have been huge settlements out of it and tons of legal costs. Does Valleywag have that kind of money? Seems to me that unless they have very deep pockets, they can’t sustain a high level of exposure forever.

btw, Brian, my daughter would be the very first one to agree with you that she doesn’t know anything.

UPS and generators take care of most power issues, data centers all over the impact area that kept on humming, so CLEARLY something else is a factor, incompetence or internal sabotage. The smoke hasn’t cleared yet, so Valleywag’s theory is still very much in play. 365Main has a history of blowouts, so I am leaning towards incompetence, as reports are falling in that their generators didn’t kick in. Regardless, I suspect the real truth will never come out.

Agreed with 15. Having been in multiple datacenters (mostly private due to the nature of my profession) but way more than Robert has, I can say that taking down a data center isn’t actually very difficult. Especially a rack. As long as you know what you’re doing, it wouldn’t take too long. In fact, there’s one data center that I know of that used to not even have redundant routers. Pull the plug on the the single point of failure, and the whole place falls.

It really depends on how the datacenter is setup and how detailed anyone would know about the internal workings and design. Not defending Valleywag, since I don’t read that … uhh… gossip, but really now…

You don’t really need to push the red button to do some serious damage if you knew how they designed their layout. We used to joke about switch techs tripping over the power cable in one of the data centers due to the design.

I, on the other hand, am defending Valleywag. It is awesome. Well, it used to be. But seriously folks, it’s fucking Valleywag. I hope they make billions of dollars, cause hundreds of heart attacks and never print a true word.

Valley wag reminds me of one of my Fathers maxims:
“Never let the truth stand in the way of a good story”.

I tend to view their articles like reading Private Eye you know they only basis they have in facts is that a journalist wrote an article.

No. Wait a second. Data centers have serious power backup systems. Power grid failures happen all the time. But not data center outages. Particularly not failures at data centers like 365Main, which has an industrial strength UPS system, batteries, and redundant diesel generators. Something’s up here, something that isn’t routine, and customers should be asking these guys some very hard questions.

LayZ: well, seeing that the journalism department at San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.