Valleywag on “drunken employee” in datacenter

Ahh, Valleywag demonstrates one more time that they’ll print any damn thing sent to them without any care whatsoever about whether or not such a thing is true or not.

First of all, if an employee wanted to take down a datacenter, they wouldn’t need to “f___ up a lot of stuff” to do it.

There’s a big red button right by the door in every data center I’ve been to. It’s an emergency power off button. It has a huge sign next to it saying that if you push it without cause you’ll be prosecuted. Why? Cause it turns off the power to everything in the datacenter. Doing that would require admins to spend hours bringing back up their equipment (like it took today).

But it’s ridiculous to say that someone could “mess up” a rack. These things are all bolted to the floor — if they weren’t a small earthquake would cause major heck. A single human being could NOT push them over, even if you were using all of your weight.

Even if someone were able to push over a rack, it hardly would take out the entire datacenter, either. And, most employees who get access to datacenters don’t have access to racks anyway. In every colo center I’ve been in (three different ones in my career) all the machines are inside locked cages. So, how would an employee get access to a cage to do enough damage to bring down at least six companies? Hint: they couldn’t.

I gotta get myself a fake email address so I can send bulls**t into Valleywag and get it printed. Jason Calacanis told me about how he got free publicity for Mahalo that way.

Anyway, I love Valleywag. The stuff that people get printed there is *funny.* Just don’t confuse it for the truth.

87 thoughts on “Valleywag on “drunken employee” in datacenter

  1. Dear Chris from BeerCoSoftware:

    You are a moron. Please switch careers; used automobile salesman is more up your alley.

    “What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit.”

    Let me give you a hint, chump: anyone interested in “enterprise, yes, or any other site with critical apps” isn’t going to accept your lowest-bid small-time crap from illbuildyourpcinmybasement.com. You don’t know the first thing about DRR or managing a datacenter.

    “We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.”

    O RLY? What happens when that “automatically shuttling network switch” runs out of power?

    Your shilling here has gotten way, way beyond reasonable. Scoble’s weblog is not your billboard for advertising your joke of a consulting company and I honestly wish he would take out your trash.

  2. Dear Chris from BeerCoSoftware:

    You are a moron. Please switch careers; used automobile salesman is more up your alley.

    “What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit.”

    Let me give you a hint, chump: anyone interested in “enterprise, yes, or any other site with critical apps” isn’t going to accept your lowest-bid small-time crap from illbuildyourpcinmybasement.com. You don’t know the first thing about DRR or managing a datacenter.

    “We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.”

    O RLY? What happens when that “automatically shuttling network switch” runs out of power?

    Your shilling here has gotten way, way beyond reasonable. Scoble’s weblog is not your billboard for advertising your joke of a consulting company and I honestly wish he would take out your trash.

  3. LayZ, your reputation must be important to you, else you wouldn’t post here anonymously.

    Given that the fake “LayZ” is your true voice, I can’t help but wonder what fake voice your real persona uses.

  4. LayZ, your reputation must be important to you, else you wouldn’t post here anonymously.

    Given that the fake “LayZ” is your true voice, I can’t help but wonder what fake voice your real persona uses.

  5. When I read the story it said nothing about the drunken employee pushing over racks; it just said the drunken employee had messed up some racks. Hitting the big red button is more likely than pushing over the racks.

  6. When I read the story it said nothing about the drunken employee pushing over racks; it just said the drunken employee had messed up some racks. Hitting the big red button is more likely than pushing over the racks.

  7. @26 “San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.”

    However, it does tell me that reputation is of no importance to them then.

  8. @26 “San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.”

    However, it does tell me that reputation is of no importance to them then.

  9. Six graders are known for ganging up on one or two classmates with lies and gossip because “it’s funny.” Woe to the kid who goes to a teacher for help, or even cries foul. He’s supposed to just sit there and take it and not disturb the fun around him.

    Nick Denton caters to people who still enjoy six grade bullying technique.

  10. Six graders are known for ganging up on one or two classmates with lies and gossip because “it’s funny.” Woe to the kid who goes to a teacher for help, or even cries foul. He’s supposed to just sit there and take it and not disturb the fun around him.

    Nick Denton caters to people who still enjoy six grade bullying technique.

  11. Scoble – holy smokes man maybe it’s because I’ve just started reading the comments but wow dude are you trolling your own comments just to create attention? Maybe you’ve only been in the giant MS/Google/Monster colo DCs and so your picture of the inside of a data center is different.

    Please listen to those of us that work in data centers day in and day out. One employee could completely destroy a DC with not a lot of effort.

    Not all DCs have an EPO switch, or at least one that’s readily available.

    Not all DCs have racks that are bolted to the floor. If they’re not bolted to the floor, you’re right it’s quite difficult to push them over, but if you open the front door and slide out a couple of servers at the top of the rack, physics does the rest.

    There aren’t always a tonne of people inside a DC – I’ve been inside a lot of different DCs at various times of the day where I am completely alone. Even if there were people, do you know many folks that would step in to physically restrain someone on a rampage? Most people shrink away from violence – I’m not sure if I would be the one to take some punches if I saw one of my coworkers obliterating racks in the DC in a drunken rage.

    Most major sites/applications/enterprises have disaster recovery plans which should is ideally instantaneous, but can take minutes. It would be entirely possible that they would be up in a couple of minutes after a major disaster and they enact their DR plan.

    I’m not suggesting that the article is correct in any way, but it certainly is plausible.

  12. Scoble – holy smokes man maybe it’s because I’ve just started reading the comments but wow dude are you trolling your own comments just to create attention? Maybe you’ve only been in the giant MS/Google/Monster colo DCs and so your picture of the inside of a data center is different.

    Please listen to those of us that work in data centers day in and day out. One employee could completely destroy a DC with not a lot of effort.

    Not all DCs have an EPO switch, or at least one that’s readily available.

    Not all DCs have racks that are bolted to the floor. If they’re not bolted to the floor, you’re right it’s quite difficult to push them over, but if you open the front door and slide out a couple of servers at the top of the rack, physics does the rest.

    There aren’t always a tonne of people inside a DC – I’ve been inside a lot of different DCs at various times of the day where I am completely alone. Even if there were people, do you know many folks that would step in to physically restrain someone on a rampage? Most people shrink away from violence – I’m not sure if I would be the one to take some punches if I saw one of my coworkers obliterating racks in the DC in a drunken rage.

    Most major sites/applications/enterprises have disaster recovery plans which should is ideally instantaneous, but can take minutes. It would be entirely possible that they would be up in a couple of minutes after a major disaster and they enact their DR plan.

    I’m not suggesting that the article is correct in any way, but it certainly is plausible.

  13. Perhaps I missed the point of this attempt at a humorous poke at a PR company. A datacenter could be downed without having to resort to knocking over a server rack and there is essentially a big red button effect. The person would have to have access of course but every data center I have been in has a reliance on power and tons of cables. There is usually a fire suppression system that can take a center down if hit and would subject the datacenter to a gas suppression. Imagine how long it would take to figure out what patch cable goes to which if a drunk/hostile/malicious person ran through a datacenter in less than 60 seconds pulling those easily accessed cables. It could cause days or weeks of outage unless you have a cohosted location.
    Just another thing to worry about… :-(

  14. Perhaps I missed the point of this attempt at a humorous poke at a PR company. A datacenter could be downed without having to resort to knocking over a server rack and there is essentially a big red button effect. The person would have to have access of course but every data center I have been in has a reliance on power and tons of cables. There is usually a fire suppression system that can take a center down if hit and would subject the datacenter to a gas suppression. Imagine how long it would take to figure out what patch cable goes to which if a drunk/hostile/malicious person ran through a datacenter in less than 60 seconds pulling those easily accessed cables. It could cause days or weeks of outage unless you have a cohosted location.
    Just another thing to worry about… :-(

  15. Sorry folks, valleywag cracks me up, their head lines read like the weekly world news that you get in the check out at the store next to all the people magazine types and gossip rag types. At least they have made themselves into a natural niche market. And they are still funny when they are not being too snarky.

  16. Sorry folks, valleywag cracks me up, their head lines read like the weekly world news that you get in the check out at the store next to all the people magazine types and gossip rag types. At least they have made themselves into a natural niche market. And they are still funny when they are not being too snarky.

  17. wreck, the bottom line is you should back up all your settings, no matter what router config or router hardware you use.

    I believe at Canix datacenter they do that.

    In our office we have a Cisco router, and a Netgear FVL328 for VPN and office routing, and the settings are backed up.

    The only thing a person could do at a datacenter before getting tackled would be to kill the master routers in the control room. If they started going into every single cage messing everything up, they would be stopped before they got very far.

    So basically that’s just 1 or 2 routers on the master to restore.

    What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit. This rack will run mirrors of all your critical services, say you have XML RPC services running through Axis or another B2B. We can make a light backup of those service to keep distributers running even when your main servers are down with switching through UltraDNS.net, or even a dedicated 3rd party DNS located somewhere super safe.
    The mirrors in Montreal will be fed with only diff data from your master in San Fransisco fed via SSL secure data transfer.
    When your datacenter in San Fran hits the floor, the software will detect it and switch the seperate DNS of all your critical business services, including but not limited to HTTP web to the Montreal location.
    While doing this for HTTP and websites only is more trivial, our service can do it for Web services like a big manufacturer that feeds datasheets to customers via ICS 650 format, xls or SOAP.
    What happens when distributers need to check the stock on items from a manufacturer and the datacenter went down?
    Those are orders for several thousands of dollars.
    Can people really afford to deal with downtime?
    We can help.

  18. wreck, the bottom line is you should back up all your settings, no matter what router config or router hardware you use.

    I believe at Canix datacenter they do that.

    In our office we have a Cisco router, and a Netgear FVL328 for VPN and office routing, and the settings are backed up.

    The only thing a person could do at a datacenter before getting tackled would be to kill the master routers in the control room. If they started going into every single cage messing everything up, they would be stopped before they got very far.

    So basically that’s just 1 or 2 routers on the master to restore.

    What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit. This rack will run mirrors of all your critical services, say you have XML RPC services running through Axis or another B2B. We can make a light backup of those service to keep distributers running even when your main servers are down with switching through UltraDNS.net, or even a dedicated 3rd party DNS located somewhere super safe.
    The mirrors in Montreal will be fed with only diff data from your master in San Fransisco fed via SSL secure data transfer.
    When your datacenter in San Fran hits the floor, the software will detect it and switch the seperate DNS of all your critical business services, including but not limited to HTTP web to the Montreal location.
    While doing this for HTTP and websites only is more trivial, our service can do it for Web services like a big manufacturer that feeds datasheets to customers via ICS 650 format, xls or SOAP.
    What happens when distributers need to check the stock on items from a manufacturer and the datacenter went down?
    Those are orders for several thousands of dollars.
    Can people really afford to deal with downtime?
    We can help.

  19. Valleywag is entertaining, and, it is a genre of journalism that predates the printing press. The notion that only hard news is journalism is a shallow one. What people like Dawn and Robert would criticize, if they remotely got the issues involved, is Valleywag’s failure to emphasize that it is usually publishing opinion, not fact. (Not necessarily honestly held, opinion, either.) But, being a clever person himself, Nick Denton likely expects his audience to figure that out for themselves.

    Ditto Brian on the rather reckless tossing around of claims of libel and slander around here. Denton knows he is on solid ground because the targets of his snarkiness (okay, lies, if you prefer) are public figures. Short of him publishing, that say, Robert Scoble buggered the bursar of San Jose State while podcasting at 365 Main, taking calls on his N95 all the while, Denton is not going to lose a libel or slander case if anyone is foolish enough to bring one. (Actually, he might get away with that example.) The flip side of being an attention seeker is that you might not like some of the attention you get.

    I have no idea what caused the problems at 365Main, but find the suggestion of a drunken employee going postal amusing. An inebriated gourmand rat having eaten through various cables would be even more entertaining. The truth will probably be boring.

  20. Valleywag is entertaining, and, it is a genre of journalism that predates the printing press. The notion that only hard news is journalism is a shallow one. What people like Dawn and Robert would criticize, if they remotely got the issues involved, is Valleywag’s failure to emphasize that it is usually publishing opinion, not fact. (Not necessarily honestly held, opinion, either.) But, being a clever person himself, Nick Denton likely expects his audience to figure that out for themselves.

    Ditto Brian on the rather reckless tossing around of claims of libel and slander around here. Denton knows he is on solid ground because the targets of his snarkiness (okay, lies, if you prefer) are public figures. Short of him publishing, that say, Robert Scoble buggered the bursar of San Jose State while podcasting at 365 Main, taking calls on his N95 all the while, Denton is not going to lose a libel or slander case if anyone is foolish enough to bring one. (Actually, he might get away with that example.) The flip side of being an attention seeker is that you might not like some of the attention you get.

    I have no idea what caused the problems at 365Main, but find the suggestion of a drunken employee going postal amusing. An inebriated gourmand rat having eaten through various cables would be even more entertaining. The truth will probably be boring.

  21. One more VC: if it really was a rogue employee there would have been LOTS of witnesses who would have talked. Have you ever been inside a datacenter during the day? I have. There’s always TONS of people inside and not just people who work for the colo either.

    And, let’s say there wasn’t any witnesses. Then there’d be tons of things left out in the open when things started going wrong. You’d see unplugged wires, destroyed machines, etc.

    They wouldn’t have been able to clean that up within minutes (which they’d have needed to do).

    Also, if someone hit the power button I’m pretty sure that works like most fire alarms and would have sprayed colored ink all over the person who pushed it, so they would have been easily identifyable.

    But, yet, no data has come out corroborating the “drunken employee” story.

    This story was made up. Pure and simple.

    And Valleywag printed it without going on scene and checking out the facts.

    Valleywag does this all the time. They write shit about me and never bother to check the facts out and never bother getting MY side of the story.

    They just love being entertaining. Valleywag +is+ entertaining, that’s for sure. But if you believe it then you’re full of it too.

    Of course, why am I arguing with a guy who isn’t a VC and won’t identify himself. I might just be arguing with Nick Denton himself here. Or maybe you’re Fake Steve Jobs? Ahh, I see your IP address doesn’t come from Boston, so that counts that out.

  22. One more VC: if it really was a rogue employee there would have been LOTS of witnesses who would have talked. Have you ever been inside a datacenter during the day? I have. There’s always TONS of people inside and not just people who work for the colo either.

    And, let’s say there wasn’t any witnesses. Then there’d be tons of things left out in the open when things started going wrong. You’d see unplugged wires, destroyed machines, etc.

    They wouldn’t have been able to clean that up within minutes (which they’d have needed to do).

    Also, if someone hit the power button I’m pretty sure that works like most fire alarms and would have sprayed colored ink all over the person who pushed it, so they would have been easily identifyable.

    But, yet, no data has come out corroborating the “drunken employee” story.

    This story was made up. Pure and simple.

    And Valleywag printed it without going on scene and checking out the facts.

    Valleywag does this all the time. They write shit about me and never bother to check the facts out and never bother getting MY side of the story.

    They just love being entertaining. Valleywag +is+ entertaining, that’s for sure. But if you believe it then you’re full of it too.

    Of course, why am I arguing with a guy who isn’t a VC and won’t identify himself. I might just be arguing with Nick Denton himself here. Or maybe you’re Fake Steve Jobs? Ahh, I see your IP address doesn’t come from Boston, so that counts that out.

  23. Something’s up here, something that isn’t routine

    Exactly. Plus, the truth is usually on the other side of a Scoble analysis. So Valleywag might not be as outlandish as it may seem.

  24. Something’s up here, something that isn’t routine

    Exactly. Plus, the truth is usually on the other side of a Scoble analysis. So Valleywag might not be as outlandish as it may seem.

  25. re chriss of hosing cisco

    “On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”

    Err recovering a bricked router is somthing your learn very early on in the CCNA course – they should have a complete set of configfiles for every router and switch ready to reload.

  26. re chriss of hosing cisco

    “On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”

    Err recovering a bricked router is somthing your learn very early on in the CCNA course – they should have a complete set of configfiles for every router and switch ready to reload.

  27. yeh But you know that those dodgy urine testing companies that frighten/bully ompanies into doing bad stuff are going to have a field day plugging testing at work for even more slaves ^h^h^h^h^h workers now :-(

  28. yeh But you know that those dodgy urine testing companies that frighten/bully ompanies into doing bad stuff are going to have a field day plugging testing at work for even more slaves ^h^h^h^h^h workers now :-(

  29. LayZ: well, seeing that the journalism department at San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.

Comments are closed.