Clearing the air with Twitter
Yesterday Evan Williams (co-founder of Twitter, his Twitter account is here) wrote me an email telling me he wasn’t blaming me and trying to clear the air. I said “can I come over?” to talk more about these issues face-to-face. I’ve always found that dealing with unpleasant topics is always better face-to-face and not over email. He wrote back and said to come on over.
I wasn’t expecting to do a video interview. I hadn’t had any sleep in 30 hours. I was tired and had just finished a dress rehearsal for our new “WorkFast.tv” show that we’re filming at Revision 3 and after that I was on Gillmor Gang, who had FriendFeed’s founders on. The show isn’t up yet, but is a very interesting one.
When I arrived at Twitter Evan Williams met me at the door (I had Twittered that I was going to be there in a few minutes). We had an off-the-record conversation which wasn’t, let’s say, fun. But we both cleared the air and then the conversation started getting interesting and I asked “can I turn on my cell phone and start broadcasting this because I think a lot of people would be interested?”
They said yes and now you can watch the rest of the conversation. I’m sorry about my goofy laugh. I was nervous and tired and it gets worse when I am both of those.
Over on TechCrunch people are giving me heck for not using a professional camera. The audio is a little rough to get. I would never have gotten this interview if I dragged around my professional camera everywhere. This wasn’t an interview opportunitity. It turned into one while we were there. Professional cameras are not appropriate things to drag along everywhere you go. So more of these interviews will be ahead and the bad audio and goofy laugh you’ll just have to deal with.
I’ll spend a while this morning to put a rough transcript of what they said here. Come back and visit this post later if you want to see what they said in text without watching the video.
It’s a 27 minute long conversation.
Outline of what we discussed, not word for word: Evan Williams and Biz Stone, co-founders of Twitter. Jesse Stay, a developer from Salt Lake and a Twitter user, was along (we were supposed to have lunch).
SCOBLE: Let’s clear the air. (This part takes the first 3:15 of interview)
EV: “For the record, we are never blaming Scoble.” They then explained how Alex Payne, a developer with Twitter, wrote a post where he was trying to explain what’s going on with the service and wasn’t trying to blame anyone.
The latest thing is that one of our developers wrote a blog post last night where he was trying to explain things, be more transparent.
EV: Explained that Scoble’s use is a “power use” of the system.
BIZ: Said one key part of the post was admitting that the service was not up to par and that the team didn’t have good enough technology in place yet to deal with the loads that Twitter is seeing.
EV: At 3:15 the discussion shifts to talk of the Instant Messaging functionality being down and why the service has been so bad in the past two weeks. He notes that in past 24 hours they’ve seen 37 minutes of downtime which, while being bad and unacceptable, is an improvement.
05:00 I note that Twitter users are extremely loyal and that even after all the downtime I’m still seeing a Tweet come in every second or two. EV discusses how fortunate Twitter is to have so many users who love the service.
05:50 BIZ talks about the plan to get out of the hole. Admits that Twitter could still have significant problems for “months.”
EV: discussed how dismaying the past two weeks have been since they had months of the service being relatively stable and had even survived the load at SXSW.
7:35 EV takes on all the people who think they have the “quick answer” noting that lots of people have told him “why don’t you get rid of Ruby already.” Notes that money isn’t the problem and that the problem is an architectural problem. At 8:24 they note that they could put me on my own server, but then no one would be able to talk with me. I thought that would be funny to many of my readers.
8:50: Jesse asks why they couldn’t open source their code base and get people outside of Twitter to help out. EV: says that won’t help, and admits that they only had four engineers who hadn’t solved this kind of problem before and that they are curing that problem now.
At this point, at about 10 minutes into the conversation, a ton of people joined the conversation and started to talk to me via Qik.com’s commenting feature. One reason why I use a cell phone is to be able to get live feedback from my audience.
10:45 I ask about the new ability for them to be able to turn off pieces of the service (right now, for instance, the XMPP gateway is turned off so we can’t use IM clients with Twitter).
EV answers that they’ve had this ability since before this year’s SXSW service and that they engineered this to be able to turn off services that are causing too much load on the core Twitter system.
13:03: I ask about how Twitter’s engine works internally and I ask if Tweets are copied for each Twitter message. For instance, do my Tweets get copied 23,000 times? EV answers that the service does NOT do that. Then talks about Twitter architecture and what they’ve learned over the last two years and how that’s showing them a path to a new architecture.
19:25 Why doesn’t Twitter stop taking new users until they fix the problems. EV says that wouldn’t make much difference.
21:50 Why doesn’t Twitter stop accounts that use scripts? EV: says that’s tough to do because Twitter has APIs and that if you turn off the people who aren’t using it in cool ways you also have to turn off the people who are using the APIs properly.
22:30 Why didn’t they build Twitter right to handle all these problems from the start? EV tells us about the history of Twitter and explains that it was built “on a lark” and that no one expected it to be a big deal. I agreed, remembering how everyone told me “Twitter is lame” first time I told them about it.
Powered By 
May 31st, 2008 at 11:17 am
Robert: What about splitting the difference and carrying around a still camera with good video capabilities? My Canon S3IS shoots optically stabilized 640×480 video with stereo audio, and given that it has a 12x optical zoom, is still pretty compact.
May 31st, 2008 at 11:25 am
Here’s another reason why I use the cell phone: because I can take questions LIVE from an audience. i can’t do that with a camcorder. I also would need to carry it around everywhere, and, sorry, I already carry enough equipment everywhere I go.
May 31st, 2008 at 11:54 am
Robert, i actually liked the video and the timing of it in response of what was going. Keep it up.
May 31st, 2008 at 11:55 am
I have only read this post about the entire meeting between Scoble and Twitter. That said, it certainly seems like Twitter is not “thinking outside the box” enough. This is not an impossible problem to solve.
As far as 37 minutes of downtime. Ha! I probably go to Twitter (an embarrassing) 10 times a day. Of those, in the last two weeks, a third of the time I get an error. Oh well, it is still cool and I wish them the best. I however am always looking for the latest and greatest. Is it friendfeed? Right now, yes.
May 31st, 2008 at 12:04 pm
I loved the video and the interview.
No matter what you do you will never make everyone happy.
May 31st, 2008 at 12:05 pm
Cool! Thanks so much for this interesting post and boiling down the conversation. I thought the video looked interesting but haven’t had time to watch it. This was very helpful.
May 31st, 2008 at 12:14 pm
So what about the quality of the video and audio. You can see it and hear it. Most of the time it is interviews like this that are the best and most relaxed. Keep them coming.
May 31st, 2008 at 12:22 pm
Robert,
I watched the video late last night and thought that it was fine.
It was impromptu, spur of the moment and refreshing.
Bringing in “professional” equipment might have stopped it from happening or curtail the discussion.
Qik is just the right medium for this type of interview/chat It takes a minute to setup and the content is more important that the quality of the video and audio. The instant viewer feedback makes it “right there right now”.
Job well done.
And, no. You did not break Twitter, it was already broken (or only half built depending what way you look at it)….
May 31st, 2008 at 12:35 pm
Great interview Robert, thanks. It was great to connect and see you and Steve in San Fran at the GaryVee book signing…hope to catch up more later.
May 31st, 2008 at 12:38 pm
You’re such a douche dude. They didn’t say anything that should have been construed as coming after you. It was very clear that they were taking responsibility and wanted to fix the problem.
May 31st, 2008 at 12:39 pm
I loved the interview and as long as I can hear what’s being said I don’t care. Why do you need HD quality video to watch an interview? Is it so that you can see every last bit of glorious action, no. When I watched the Qik video, I was also browsing and doing other things, I didn’t have to see what was happening to know what you were saying. It’s ridiculous to complain about the quality of video of an interview…
May 31st, 2008 at 12:41 pm
Hey Robert,
Thought Qik worked just fine. I prefer impromptu interviews. They are more real and in the moment.
Keep up the great work.
-Paul.
May 31st, 2008 at 1:02 pm
I try and watch the bad video and audio (no just on this interview), but it’s too much, can’t watch it.
You can’t carry a Canon HV20 that shoots HD? and fits in the palm of your hand?
I shoot my 10 month old with this camera … maybe I expect a professional interviewer to be more professional than I am with my home movies.
May 31st, 2008 at 1:10 pm
Robert, you should NEVER go so long without sleep. In 1995 a good friend of mine dropped dead from a heart attack at the age of 47. He had just been through a battery of physicals, including stress tests and EKGs, because he was returning to active duty. They said his heart was in tiptop shape, despite his beer belly.
So why did he have a heart attack? Because he went three days with very little sleep. He was trying to get lots of things done so his wife wouldn’t have problems while he was deployed. Turns out his parting was forever.
Maryam is going to be totally pissed at you if you drop dead from working so hard, just like I’m angry at my husband for dying from skin cancer at age 46 when I used to BEG him to wear sunscreen. Me: “Fine!, but don’t come crying to me when you get skin cancer.” Him: “I’m not going to get skin cancer.” Yeah, right. And you can’t have a heart attack either, right Robert?
I know you love to work, that your work is also your play, but you’re going to have to stop this manic pace you’ve been keeping before it permanently stops you. You’re “on” all the time. You need to turn OFF, *at least* 8 hours EVERY day.
I’m only nagging you because I don’t want to see Maryam and Patrick and Milan go through what me and my two kids have gone through. Please don’t do that to them.
May 31st, 2008 at 1:10 pm
>You can’t carry a Canon HV20 that shoots HD? and fits in the palm of your hand?
No. I already carry enough equipment around, thanks. And that one can’t post to the Web without additional equipment and connections. It also can’t hold a live chat room so your audience can participate.
I have $6,000 HD cameras for my show on http://www.fastcompany.tv but they need a separate person to both carry them, set them up, make sure they are set right (they require things like white balancing, microphone tests, and other stuff) and when you use two cameras you’ve also gotta do a lot of post-processing work. The picture and audio look like CNN or Discovery Channel (they use the same equipment) but it takes days/weeks to get that stuff up.
I far prefer doing live interviews with people who are interactive. I also want to be able to get my videos up while people care, not five hours later. In fact, if I had used a camcorder yesterday I still wouldn’t have been able to get the videos up (after the interviews yesterday I had to drive to pick up Patrick, then was dead tired, and only got up this morning and today’s a weekend day so I don’t do real work, just fun stuff like blogging and FriendFeeding).
May 31st, 2008 at 1:11 pm
Dawn: good point. I got lots of sleep last night. The night before was stupid.
May 31st, 2008 at 1:41 pm
I wanted you to know that I thought the interview was timely and done well considering that it was a last minute decision. Rather then focus on you not having the equipment to set up a “professional” meeting, I think the main focus should be how quickly we can connect socially with even the smallest of equipment pieces, a cell phone. Was this as readily available with ease of use five years ago? Ten years ago?
Keep up the great work and keep keeping us informed!
May 31st, 2008 at 2:11 pm
I never cease to be awed by the depth of information and argument you get to in the shortest of time. Thanks for your reporting. Good to see someone who is so open to criticism and doles it out in a purely constructive manner.
What twitter is concerned: it’s good to know that they are aware of their problems (duh) and are open to communication. I think the really important point, which might save twitter in the long run, is the deserved loyalty by the base.
Stick to a good thing.
May 31st, 2008 at 2:16 pm
Cool - I thought they were going to ration your tweets!
May 31st, 2008 at 2:47 pm
Scoble, you’re awesome. I don’t really care about your clearing the air (no offense to you) I just really like your reporting. I am just amazed at the excuses coming out of Twitter. They’re snatching defeat from the jaws of victory. Anyone who says “there will be problems for some time to come” should be fired. The right people with the right leadership with the right funding can fix anything. I suspect they have the right people and the right funding.
May 31st, 2008 at 2:51 pm
Watched the “conversation” yesterday and it was quite useful to understand where Twitter is right now and love the summary, helpful to refer to those who don’t understand Qik social live streaming value and just want the facts.
Looks like at the beginning, Twitter was really just a microblogging platform and the problems with integrating the whole potential a messaging protocol (XMPP) can provide to the infrastructure are the architectural problems they are facing. I was surprised that they mentioned they only have one copy of the data in the disk with multiple caching updates in memory, they really have pushed the limits with this architecture and the move to a messaging platform it’s going to help the downtime, hope it doesn’t take them too long.
There is a opportunity for the appearance of a WordPress in microblogging (maybe a FriendFeed or Grazr open source). And for messaging, the fact that FriendFeed looks email as their competition (i.e. rooms=RSS mailing lists) signals they have already thought beyond microblogging. If more applications are built related to the mobile web like Moblf.com, then Robert is right and Twitter loyal users can look FF as a viable option.
May 31st, 2008 at 4:18 pm
[...] Scoble recently recorded a spur-of-the-moment interview with the founders of Twitter. As he notes in his blog post, some folks are giving him grief for not [...]
May 31st, 2008 at 5:02 pm
how everyone told me “Twitter is lame” first time I told them about it
And the second, and the third…to eternal infinitude, no matter, it’s still lame.
“On a lark”, heh, which pretty much sums up most of what passes for “Web 2.0″…
You do know they have Sony Handheld HD’s, something like the old HDR-HC1, you could get a basic Steadicam Merlin to go with, and dump to SD, and x264 it pretty quick, maybe even using some form of hardware rendering, keep time limit to 20 min or under, and with the Merlin, you might not need to edit it much, posting fairly raw. Quick and professional, with the HDR-HC1 you can shoulder-bag carry, being quite inconspicuous. Cell phone video are just headaches, no matter what the excuse.
May 31st, 2008 at 5:07 pm
Christopher: well, get some aspirin.
My camera can broadcast live, which is a MAJOR advantage over what you’re talking about, especially if you want a two-way audience like I do. And I don’t even need to find an Internet connection to upload, either.
May 31st, 2008 at 5:28 pm
Hi Robert,
Why don’t you use a Flip Ultra camera? (http://www.theflip.com/). I’m told the quality is quite high and it seems to be very portable.
BTW, I don’t have anything to do with The Flip company or any other company related to the cameras world.
Greetings from Barcelona, Spain.
May 31st, 2008 at 5:38 pm
Jesse Stay is the Book Author of O’Reilly “Facebook Essential” and Co-Author of “I’m on Facebook — Now What?”. Jesse also runs the Facebook Developer Garage - Salt Lake City Utah. He is the co-organizer of the Western Region of OpenSocial Hackathon Episode III along with Silicon Valley Web Builder (SVWB). Jesse is not just a developer!
I would have make sure this Scoble to take power snap during his 30 hr working around the clock schedule. Jesse has 4 kids including a new born. We wouldn’t want to see 2 super dad to get into any silly auto accident.
I suggest we put an automatic warning if any blogger twitter nonstop for 24 hrs. We shut off the Twitter feature to put them to sleep. This would be a safety feature and family-protection plan. Many families will be supporting this feature.
@Dawn Douglass health warning is good one!
May 31st, 2008 at 5:41 pm
Robert,
Your interview was great. I thought you asked good questions and was glad to see both Biz and Evan there.
About the quality of video–Qik isn’t meant to have the best video quality available (yet?). It’s meant to capture video on-the-go while giving you a channel to interact with your audience.
That’s exactly what you used it for here.
Anyone suggesting you used a “regular” video camera doesn’t “get” Qik.
Don’t listen to the nay-sayers–keep doing what you’re doing Robert.
- Brad
May 31st, 2008 at 5:45 pm
So basically the end result is that you and your ego were overreacting to what Twitter posted.
May 31st, 2008 at 5:56 pm
All the goofs worrying about the process instead of the content. Probably won’t watch black and white films, either.
Good job getting the story and getting it out quickly.
May 31st, 2008 at 6:24 pm
Maybe it would have helped if you had gone to journalism school, before they handed you a cell phone with a camera. Your no Dan Rather, that’s for sure.
Dump the stupid laugh, and buy a tripod. And maybe one day they’ll buy you a real camera. Goof Ball.
May 31st, 2008 at 6:41 pm
I think twitter is cool, and so far I can stand an ocassional twiccup.
I think Twitters problems is partly caused by the many subscribers it has, but also on the dozens of great 3rd party applications that also puts a load in its db. I agree that twitters notwithstanding its problems, twitter should not cut them off.
I think so far friendfeed is not quite there, and it is lame. I also don’t understand your enthusiasm there.
When Friendfeed succeeds, you can quote me on that (grin)….
May 31st, 2008 at 6:46 pm
The best part, in a terrific interview, is the shot of Scoble’s Friendfeed T-Shirt while in the Twitter office!
All seems well for the moment in the Web 2.0 world (that is a good thing).
Now over to Gillmor Gang to listen to the just released show with Friendfeed founders Bret Taylor (genius ++) and Paul B. (he may be as well).
May 31st, 2008 at 7:47 pm
[...] at Twitter invited Scoble over to “clear the air”. Read Scoble’s post, transcript, and video here. Now Michael Arrington is calling Twitter out with some questions of his own – which are pretty [...]
May 31st, 2008 at 10:13 pm
Beth, ROFL! I could tell Robert was getting a little tired towards the end there. I was pretty exhausted myself. :-)
May 31st, 2008 at 10:14 pm
Sorry, that was to Bess, not Beth.
May 31st, 2008 at 11:12 pm
Robert,
What is your professional camera?
June 1st, 2008 at 2:30 am
Twitter is a hard one to figure out in terms of its mainstream acceptability. Its big in geek land but most people outside “our” inner circle just don’t get it.
June 1st, 2008 at 4:45 am
Wilson ng:
I think so far friendfeed is not quite there, and it is lame. I also don’t understand your enthusiasm there.
Can you elaborate on why it is you think FF is lame? What is missing? What is flawed? Aren’t they reacting quickly enough to recommendations from the base? Or what, exactly?
June 1st, 2008 at 8:04 am
In the era, of TIVO, Archos TV, Xvid and other misc. time-shifting, the “advantages” of being “live” are minimal, so maybe you get that audience feedback from people who just happened to be glued to their LCDs, at any given time, but once that fleeting moment is gone, the output is, well, junk.
Same thing with moblogging, people posting their lameo out of focus photos inrealtimegeeisntthiscool, but what happened? That “meme”, died hard and quick, with the end result of people started posting good pictures, not live, but good pictures on pay presentational sites, of course, the mainstream had already been doing that since the birth of the net, but it took Flickr for the geeks to get all flubbered. Not live, but good. The way you are doing it, the value is lost the second it’s over.
June 1st, 2008 at 9:16 am
Hi Robert, great opportunity that you had and well catched despite the laughs ;-). About the audio I use with my N95 a Plantronics P590. It has two advantage : 1/ on a “duo” i put the headset round the neck of the interwiewee and thru the bluetooth, i’ll get a better than 8k builtin mic.
and it’s very efficient on a very noisy area like in a street..
On a group of discussion like the one that you had at twitter offices , I usually put the headset on a table and the mic seems to do a good job.
you can see example of my (french) interveiws on http://blugture.blip.tv
I realize talking this way about a product is likely considered as advertisement, if anyone have a clue to how talk about a product you like without beiing considered as a marketer or whatever devil ppl, please help me out .. :-)
June 1st, 2008 at 9:24 am
[...] your repair process will take months. To prevent user defections en masse, hone your communication skills to a perfect, fevered pitch. [...]
June 1st, 2008 at 9:28 am
[...] A question to add to all of these posts asking Twitter questions: [...]
June 1st, 2008 at 9:56 am
How timely… and there’s even a scobleizer cameo in my new video!
check it: http://www.youtube.com/watch?v=93dGW_hDuQ0
June 1st, 2008 at 10:36 am
[...] then goes over and has a conversation with Evan Williams, co-founder of twitter. Scoble writes: Yesterday Evan Williams (co-founder of Twitter, his Twitter [...]
June 1st, 2008 at 1:09 pm
Hey Jesse
Didn’t know you have so much fun the next day - landing an exclusive Twitter interview that Tech Crunch has to write about it.
Scoble has been good about resting. At least after my post, he didn’t stay on to monitor the blog post.
That Mike also likes to Twitter non-stop around the clock. He can afford it because he is single. His staff can continue his legacy.
I have another brilliant Architecture idea. We should advise Twitter to purposely turn down the service or fail the server whenever bloggers Twitter over 24 hrs :)
June 1st, 2008 at 1:30 pm
Robert- Do you ever have moments when you sit back and laugh while thinking… “Wow. I actually AM kind of a big deal.” If not… you really should.
June 1st, 2008 at 1:35 pm
Robyn: we’re all a big deal.
June 1st, 2008 at 6:44 pm
[...] verdade não era bem isso, o que Scoble conseguiu esclarecer com os caras e serviu pra acalmar os ânimos do mercado e da comunidade de [...]
June 2nd, 2008 at 1:51 am
[...] “I wonder if Scoble could be used as a scalability term. “This application support five 9’s and is fully [...]
June 2nd, 2008 at 8:52 am
[...] popular microblogging platform–is down for the count (something that’s been happening a lot lately), they display this charming image on their [...]
June 2nd, 2008 at 11:07 am
[...] użytkowników o problemy jakie mają. Twitter szybko jednak rozwiązał problem zapraszając przewrażliwionego Scoble do siebie i wyjaśniając całą [...]
June 2nd, 2008 at 6:44 pm
Good interview Robert. Qik works just fine for this type of impromptu format. Good to see that the Twitter guys are humbly addressing the problem.
June 2nd, 2008 at 8:33 pm
[...] NOTE: The Twitter folks claim they don’t copy the tweet around for each follower. I’m sure they don’t. But they don’t say anything about copying the ID around (which, as I’ve stated, makes a good amount of sense if you were architecting Twitter 18 months ago). 13:03: I ask about how Twitter’s engine works internally and I ask if Tweets are copied for each Twitter message. For instance, do my Tweets get copied 23,000 times? EV answers that the service does NOT do that. (scobleizer.com [...]
June 3rd, 2008 at 12:37 pm
[...] companies/organisations how to “fix” a problem. There has been a huge amount of commentary about twitter, and so why not throw in my own 2 cents (or pence/euro’s [...]
June 5th, 2008 at 6:44 pm
[...] Qik user (he recently used the service for an important interview with Twitter’s founders, defending himself by saying that he would have never been able to get the interview in the first place had he [...]
June 6th, 2008 at 11:46 am
[...] Qik user: He recently used the service for an important interview with Twitter’s founders, defending himself by saying that he would have never been able to get the interview in the first place had he [...]
June 13th, 2008 at 9:02 am
[...] folks may have read about the tiff a while back between Robert Scoble and Twitter. Scoble did meet with Twitter and resolve the situation. Some folks are speaking about a business model with these micro-blogging [...]
June 13th, 2008 at 12:03 pm
[...] the material is interesting and happy to read, it absolutely keeps you addicted. Best part is that you can view without being forced to sign-up. [...]
June 17th, 2008 at 1:40 am
[...] folks may have read about the tiff a while back between Robert Scoble and Twitter. Scoble did meet with Twitter and resolve the situation. Some folks are speaking about a business model with these micro-blogging [...]
June 22nd, 2008 at 12:54 pm
[...] its user base disappears ).Whatever happens, then thing I find so hard to believe is how much some people get wound up about this “fun” new concept… is it really that critical to your [...]
June 22nd, 2008 at 8:44 pm
[...] 在Read/Write Web上也看到這一篇Can Twitter Be Saved?Twitter請了Pivotal實驗室來幫助他們解決架構上的問題。後來在Robert Scoble的訪問，Twitter也談到了他們所碰到的架構擴展問題。 [...]
August 14th, 2008 at 11:54 pm
[...] business, write 2 books, join the Utah v100 list of top Entrepreneurs, get on Techcrunch (3 times), Scobleizer, Webware, LouisGray, AllFacebook, InsideFacebook, Guy Kawasaki, and other blogs, as well as become [...]