Ben and Mena Trott say they are sorry

If you aren’t a geek who has to keep a data center running you might not appreciate the post that Mena Trott, co-founder of Six Apart, made yesterday. The business challenges facing all of us are deep and extreme as we deal with a world that doubles in size every few months (and for some services is on a far far steeper growth curve).

When you hear about employees leaving big and established companies for smaller startups, you don’t often hear about the long nights, the very real risk of failure. Silicon Valley is paved with the failures of entrepreneurs.

One thing, though, is that I really appreciate management that gives me the straight up unvarnished facts. That’s the kind of people I want to work for (and with). It’s the kind of people I want to bet my own business on.

Hey, Mena, remember eBay? They had similarly horrible days and went on to build a great business. Please do keep bringing us inside your company and letting us know how things are going.

19 thoughts on “Ben and Mena Trott say they are sorry

  1. Robert:

    Thanks for the post and the blog.

    You write:

    “If you aren’t a geek who has to keep a data
    center running you might not appreciate the
    post that Mena Trott, co-founder of Six Apart,
    made yesterday.”

    I’m an empathetic contrarian. There are a good number of geeks out there who keep much larger operations running 24×7. It’s certainly important to point out the value to customer relations of Ms. Trott’s public note of Six Apart’s recent shortcomings, but it’s equally important for those of us who are responsible for infrastructure to point out that these are neither unforeseeable nor insoluble problems.

    The type of problem that Six Apart now experiences (and eBay and countless others before them have also felt) is a fully preventable problem. But only if there is a deep committment to consider the area of infrastructure as important a part of company growth philosophy as the areas of staffing, revenue capture, and customer retention.

    Too many companies make the risk-based tradeoff that they can shoot up the hockey stick-like customer growth curve while gambling on the infrastructure that got them out of the gate.

    In some cases, the tradeoff works. (Perhaps because the anticipated growth never materializes, perhaps because of sheer good luck in early infrastructure buildout.) Sadly, in too many cases, it doesn’t.

    And lest you read only ‘hardware’ into the word infrastructure, I use it to mean many broader things.

    Clearly individual servers, mass storage, local networks, and external access pipes are all very, very important. Getting the right ones to begin with (and putting them together the right way) would be a coup for most startups. Many fail at this step because they feel an inordinate need to bootstrap at startup with systems and networks that they purchase from Best Buy or the like.

    But in addition to focusing on server, storage, and network elements, the core architects of the software need to be thinking out of the gate about how their apps should be built for distributed access, redundancy, and seamless failover. Even in 2005, too many people are still building v1.0 apps that only work on one system, or one network, or one data center.

    And the discussions between technical and executive management should never unwittingly devolve into the tradeoff-type discussions that happen too commonly in tight budget situations.

    How many of you have been on one side or the other of this discussion?

    CEO: We’re rolling out our new app next week. We expect a spike in traffic. Can our network handle it?
    TECH: I don’t know. You promised we could upgrade our routers after our last release but you still haven’t authorized the purchase.
    CEO: Listen, the budget won’t allow for both the marketing for this release and new hardware. Plus we don’t have time to delay this release. So just make it work. We’ll get you the new routers when we close the next round.

    Well guess what happens? The app is released. The spike materializes. The infrastructure fails in mysterious ways. You cobble together short-term adjustments to get back up and running, while thinking about medium-term solutions for the root causes of these problems, all the while the data growth curve is already shooting past your best current estimates.

    Now emergency budget allocations are made to buy more hardware to throw at the problem, but it’s too little in some areas, too much in others, and poorly stitched together because you can’t architect while firefighting.

    Also, sadly, the software can’t be reengineered for scalability as quickly as you can slot in more hardware. So you may have staved off the problem for a little while, but in your gut you know that the levee is just days away from breaching again, and you’ll soon be flooded under another 3TB of data.

    And your poor CEO will have to make another public apology, and your exhausted CTO and his techs will be spending another week of 18-hour days in his data center wondering just why in the hell their Cisco 7606 isn’t passing 30 million packets per second and their front-end and app servers have to be rebooted every thirty minutes.

    Welcome to Web 2.0, same as Web 1.0.

  2. Robert:

    Thanks for the post and the blog.

    You write:

    “If you aren’t a geek who has to keep a data
    center running you might not appreciate the
    post that Mena Trott, co-founder of Six Apart,
    made yesterday.”

    I’m an empathetic contrarian. There are a good number of geeks out there who keep much larger operations running 24×7. It’s certainly important to point out the value to customer relations of Ms. Trott’s public note of Six Apart’s recent shortcomings, but it’s equally important for those of us who are responsible for infrastructure to point out that these are neither unforeseeable nor insoluble problems.

    The type of problem that Six Apart now experiences (and eBay and countless others before them have also felt) is a fully preventable problem. But only if there is a deep committment to consider the area of infrastructure as important a part of company growth philosophy as the areas of staffing, revenue capture, and customer retention.

    Too many companies make the risk-based tradeoff that they can shoot up the hockey stick-like customer growth curve while gambling on the infrastructure that got them out of the gate.

    In some cases, the tradeoff works. (Perhaps because the anticipated growth never materializes, perhaps because of sheer good luck in early infrastructure buildout.) Sadly, in too many cases, it doesn’t.

    And lest you read only ‘hardware’ into the word infrastructure, I use it to mean many broader things.

    Clearly individual servers, mass storage, local networks, and external access pipes are all very, very important. Getting the right ones to begin with (and putting them together the right way) would be a coup for most startups. Many fail at this step because they feel an inordinate need to bootstrap at startup with systems and networks that they purchase from Best Buy or the like.

    But in addition to focusing on server, storage, and network elements, the core architects of the software need to be thinking out of the gate about how their apps should be built for distributed access, redundancy, and seamless failover. Even in 2005, too many people are still building v1.0 apps that only work on one system, or one network, or one data center.

    And the discussions between technical and executive management should never unwittingly devolve into the tradeoff-type discussions that happen too commonly in tight budget situations.

    How many of you have been on one side or the other of this discussion?

    CEO: We’re rolling out our new app next week. We expect a spike in traffic. Can our network handle it?
    TECH: I don’t know. You promised we could upgrade our routers after our last release but you still haven’t authorized the purchase.
    CEO: Listen, the budget won’t allow for both the marketing for this release and new hardware. Plus we don’t have time to delay this release. So just make it work. We’ll get you the new routers when we close the next round.

    Well guess what happens? The app is released. The spike materializes. The infrastructure fails in mysterious ways. You cobble together short-term adjustments to get back up and running, while thinking about medium-term solutions for the root causes of these problems, all the while the data growth curve is already shooting past your best current estimates.

    Now emergency budget allocations are made to buy more hardware to throw at the problem, but it’s too little in some areas, too much in others, and poorly stitched together because you can’t architect while firefighting.

    Also, sadly, the software can’t be reengineered for scalability as quickly as you can slot in more hardware. So you may have staved off the problem for a little while, but in your gut you know that the levee is just days away from breaching again, and you’ll soon be flooded under another 3TB of data.

    And your poor CEO will have to make another public apology, and your exhausted CTO and his techs will be spending another week of 18-hour days in his data center wondering just why in the hell their Cisco 7606 isn’t passing 30 million packets per second and their front-end and app servers have to be rebooted every thirty minutes.

    Welcome to Web 2.0, same as Web 1.0.

  3. Pingback: Elliott Back
  4. Robert –

    FYI — all Typepad customers PAY (and not a small amount) for their service. While an apology (under duress, I might add) is nice, it’s not enough. I’m on the verge of switching (it’s not like this is the first time this has happened).

    FMF

  5. Robert –

    FYI — all Typepad customers PAY (and not a small amount) for their service. While an apology (under duress, I might add) is nice, it’s not enough. I’m on the verge of switching (it’s not like this is the first time this has happened).

    FMF

  6. Christopher you made me laugh so hard I am crying. Cute as a bug is right. I also have a TypePad account that has been unbearable in the last few weeks. It is painful to use. I find it interesting that the apology lacks a payback particularly when on October 6th, SixApart lost data to some of their blogs beginning with an “S”. When that happened they issued a free 6 month extension to Steve Rubel for his inconvienance.

    http://www.micropersuasion.com/2005/10/typepad_loses_s.html

    Different circumstances yes, but of the same cloth. Where is my salve?

  7. Christopher you made me laugh so hard I am crying. Cute as a bug is right. I also have a TypePad account that has been unbearable in the last few weeks. It is painful to use. I find it interesting that the apology lacks a payback particularly when on October 6th, SixApart lost data to some of their blogs beginning with an “S”. When that happened they issued a free 6 month extension to Steve Rubel for his inconvienance.

    http://www.micropersuasion.com/2005/10/typepad_loses_s.html

    Different circumstances yes, but of the same cloth. Where is my salve?

  8. A preview…

    Oh heck, we’ve just seen a failure in a piece of networking equipment that had never failed before, and so on, ranging from hardware failures to software failures. We are cute. We are cute. We need more money. We apologize for the poor service you’ve experienced over the past four years, and also for the lack of official communication from our blogs. We need someone who knows how to run an actual company. We’ve seen failures in our storage servers. We’ve seen failures in our software. We’ve seen failures in our hardware. We’ve seen failures in our networking equipment. There are a number of big issues that have always bothered us about all of our products. But we are still cute. We are cute. We need more money. Some people have the innate ability to consistently write wonderfully, but for the rest of you, we have a whole menu section of blogs. We are cute. We are cute. It’s quite unbelievable that we’ve been doing this for four years. This has been a bad year for TypePad’s performance and general availability, and we’d like to talk about a number of the issues we’ve faced, how frustrated they make us. But never mind that, check out Project Doomed Comet. Oh this running a company is hard. I like Westies. Building Speill Checick and WYSIWYG is a lot harder than it seems. We can’t scale. But we are still cute. I mean just look at us. Cute as a bug.

  9. A preview…

    Oh heck, we’ve just seen a failure in a piece of networking equipment that had never failed before, and so on, ranging from hardware failures to software failures. We are cute. We are cute. We need more money. We apologize for the poor service you’ve experienced over the past four years, and also for the lack of official communication from our blogs. We need someone who knows how to run an actual company. We’ve seen failures in our storage servers. We’ve seen failures in our software. We’ve seen failures in our hardware. We’ve seen failures in our networking equipment. There are a number of big issues that have always bothered us about all of our products. But we are still cute. We are cute. We need more money. Some people have the innate ability to consistently write wonderfully, but for the rest of you, we have a whole menu section of blogs. We are cute. We are cute. It’s quite unbelievable that we’ve been doing this for four years. This has been a bad year for TypePad’s performance and general availability, and we’d like to talk about a number of the issues we’ve faced, how frustrated they make us. But never mind that, check out Project Doomed Comet. Oh this running a company is hard. I like Westies. Building Speill Checick and WYSIWYG is a lot harder than it seems. We can’t scale. But we are still cute. I mean just look at us. Cute as a bug.

  10. What seems missing in that apology is a payback to the users. It would have been so much more well received if users got back something as compensation than just words (heartfelt I agree), say a month free.

  11. What seems missing in that apology is a payback to the users. It would have been so much more well received if users got back something as compensation than just words (heartfelt I agree), say a month free.

  12. Paul: even if you have Microsoft’s resources it’s hard to keep up with the rapid growth that this industry is seeing.

    In fact, Google’s employees were complaining to me about their inability to keep up with the growth they are seeing.

  13. Paul: even if you have Microsoft’s resources it’s hard to keep up with the rapid growth that this industry is seeing.

    In fact, Google’s employees were complaining to me about their inability to keep up with the growth they are seeing.

  14. Six Apart got 10 Million $$ in VC funding and you would think they could have anticipated growth and ramped up their servers.

  15. Six Apart got 10 Million $$ in VC funding and you would think they could have anticipated growth and ramped up their servers.

Comments are closed.