Should services charge “super users”?

Om Malik says that Twitter should charge super users like me and come up with a business model.

Dare Obasanjo, in a separate, but similar post comes to the conclusion that Twitter’s problems are due to super users like me.

Interesting that both of these guys are wrong.

First of all, Twitter doesn’t store my Tweets 25,000 times. It stores them once and then it remixes them. This is like saying that Exchange stores each email once for each user. That’s totally not true and shows a lack of understanding how these things work internally.

Second of all, why can FriendFeed keep up with the ever increasing load? I have 10,945 friends on FriendFeed (all added in the past three months, which is MUCH faster growth than Twitter had) and it’s staying up just fine.

But to the point, why not charge super users? I’d pay. But, if Dare and Om are right, there’s no way that I’d support the service enough to pay for my real cost on the service.

Either way, Twitter’s woes were happening long before my account got super huge. Remember SXSW last year? I only had 500 followers and Leo Laporte had something like 800. The service still went down. If this were a straight “n-scale” problem the crashing problems wouldn’t have shown up so early.

Why not just limit account size, like Facebook did? Well, that’s one way to deal with the problem, but if you look at my usage of Facebook it’s gone down to only a few minutes every month. I don’t even answer messages there anymore. Why? Cause I get frustrated at getting messages from people who wonder why I won’t accept them as a friend. It’s no business “utility” if I can’t make infinitely large friend lists and use those lists in the same way I use email (which Facebook also bans).

So, what do I do? I get excited by FriendFeed which lets 11,000 people interact with me in a public way. I have a feeling that that rapid growth will continue unabated and so far Friendfeed has stayed “Google fast.”

Nice try, though.

139 thoughts on “Should services charge “super users”?

  1. I just wanted to put in another “Everyone should read Nick Halstead’s comment” vote. Honestly, there are many things I admire about Scoble (getting the obligatory compliment before the insult out of the way) but this post is just ridiculous from at technical perspective.

    As far as “charging super users” goes it isn’t really worth arguing because its going to be different for every service.

    This is why you need a business model. To determine which ways of making money will be most effective and execute on them. Charging super users will be right in some cases while being wrong in others (depending on how much value the company in question can put into the “charged” scenario)

  2. I just wanted to put in another “Everyone should read Nick Halstead’s comment” vote. Honestly, there are many things I admire about Scoble (getting the obligatory compliment before the insult out of the way) but this post is just ridiculous from at technical perspective.

    As far as “charging super users” goes it isn’t really worth arguing because its going to be different for every service.

    This is why you need a business model. To determine which ways of making money will be most effective and execute on them. Charging super users will be right in some cases while being wrong in others (depending on how much value the company in question can put into the “charged” scenario)

  3. Everyone should read Nick Halstead’s comment.

    Robert, I’m sure from reading these comments that the people talking about the technical problems understand how a the normalized databases they teach you in Computer Science course work. It just that large system can’t use them (flickr for example doesn’t it’s sharded / de-normalized)

    I don’t think is twitter is sharded yet since they weren’t at 350,000 users (http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)
    They certainly SHOULD be copying messages around if they are sharded.

    You would think if they could get to 350,000 users on one database they could get to 1 million users by adding some database read-only slave servers.

    Scaling isn’t about saving disk space, CPU cycles, memory – that is being efficient it’s not the same thing. Microsoft might try that with Exchange to reduce their customers hardware costs (not that it works from what I hear)

    Scaling is knowing you can buy a rack of machines of servers and actually make them reduce your load.

  4. Everyone should read Nick Halstead’s comment.

    Robert, I’m sure from reading these comments that the people talking about the technical problems understand how a the normalized databases they teach you in Computer Science course work. It just that large system can’t use them (flickr for example doesn’t it’s sharded / de-normalized)

    I don’t think is twitter is sharded yet since they weren’t at 350,000 users (http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)
    They certainly SHOULD be copying messages around if they are sharded.

    You would think if they could get to 350,000 users on one database they could get to 1 million users by adding some database read-only slave servers.

    Scaling isn’t about saving disk space, CPU cycles, memory – that is being efficient it’s not the same thing. Microsoft might try that with Exchange to reduce their customers hardware costs (not that it works from what I hear)

    Scaling is knowing you can buy a rack of machines of servers and actually make them reduce your load.

  5. First, as someone has already noted, any large e-mail system will indeed duplicate messages being sent to a large number of recipients — typically a single copy per server, with each recipient getting a Unix hard link (or equivalent) to that copy.

    There are several critical differences between Twitter and e-mail, however. The push or notification aspect is one, but message size is a big one. In particular, each of the hard links pointing to a single instance of an e-mail will be bigger than the entire body of a Tweet! Duplicating messages, even in pathological cases like Scoble’s, is trivial: 25,000 copies of a 140 byte message represents a mere 3.5 Mbytes, smaller than a single large e-mail body!

    Similarly, I think you’re overestimating the burden of keeping pre-calculated per-viewer data around: the default view has about 16 messages, each 140 bytes plus a bit of metadata (sender username/icon URL), total perhaps 3.2K. 10,000 users on the server? 32 Mb! Trivial. Even ten *million* users on a single node would fit on a PC you can buy online from Dell!

    The best architecture is probably a hybrid: keep the recent message queue in RAM for active users (and update realtime when those they follow post messages), built the cache from disk when they log in. Even on a single host, with 15kRPM drives (4ms writes), that’s 100 spindle-seconds; a pair of Apple’s 16-drive arrays and you’re looking at three seconds to process a Scoble-tweet, ignoring both write merging and RAID overhead.

    In reality, of course, you can omit a lot of those write-barriers and re-issue the writes from a redo log in the event of a crash, cutting the write load still further. Mirror the writes and distribute reads consistently, you get failover and gain cache hits to boot (each server only sees half as many active users).

    Or you write it all in Ruby and SQL then throw a kajillion dollars worth of hardware at making it all sort of work most of the time through brute force. Even $15m can only buy you so much brute force, though…

  6. First, as someone has already noted, any large e-mail system will indeed duplicate messages being sent to a large number of recipients — typically a single copy per server, with each recipient getting a Unix hard link (or equivalent) to that copy.

    There are several critical differences between Twitter and e-mail, however. The push or notification aspect is one, but message size is a big one. In particular, each of the hard links pointing to a single instance of an e-mail will be bigger than the entire body of a Tweet! Duplicating messages, even in pathological cases like Scoble’s, is trivial: 25,000 copies of a 140 byte message represents a mere 3.5 Mbytes, smaller than a single large e-mail body!

    Similarly, I think you’re overestimating the burden of keeping pre-calculated per-viewer data around: the default view has about 16 messages, each 140 bytes plus a bit of metadata (sender username/icon URL), total perhaps 3.2K. 10,000 users on the server? 32 Mb! Trivial. Even ten *million* users on a single node would fit on a PC you can buy online from Dell!

    The best architecture is probably a hybrid: keep the recent message queue in RAM for active users (and update realtime when those they follow post messages), built the cache from disk when they log in. Even on a single host, with 15kRPM drives (4ms writes), that’s 100 spindle-seconds; a pair of Apple’s 16-drive arrays and you’re looking at three seconds to process a Scoble-tweet, ignoring both write merging and RAID overhead.

    In reality, of course, you can omit a lot of those write-barriers and re-issue the writes from a redo log in the event of a crash, cutting the write load still further. Mirror the writes and distribute reads consistently, you get failover and gain cache hits to boot (each server only sees half as many active users).

    Or you write it all in Ruby and SQL then throw a kajillion dollars worth of hardware at making it all sort of work most of the time through brute force. Even $15m can only buy you so much brute force, though…

  7. Maybe you’re both right. Maybe it keeps a single copy of each tweet text and copies a tweet ID to each user’s queue. The heavy-lifting of building the queue would be done at write time. To build a page it would look up each tweet ID in a user queue using a simple key-value map (which can easily be replicated and scaled.)

  8. Maybe you’re both right. Maybe it keeps a single copy of each tweet text and copies a tweet ID to each user’s queue. The heavy-lifting of building the queue would be done at write time. To build a page it would look up each tweet ID in a user queue using a simple key-value map (which can easily be replicated and scaled.)

  9. Trevor: well, if every Tweet is copied for every one of its potential readers, that totally explains why Twitter has some scale problems. Most Twitter users don’t use the service very often, if at all (I watch).

    In my scenario there ARE copies. Just not automatic ones. Also, Twitter only needs to keep the last 10 Tweets cached on each user’s page, to keep the home page fast. Other pages take forever to load, so I doubt those are cached. Even in the home page scenario my Tweets would only be copied to those users who haven’t had my Tweets replaced by other users (most of the time my Tweets would be pushed lower, so there wouldn’t be 23,000 copies, only, maybe 1,000).

    Either way, if I’m to blame for Twitter going down, why isn’t FriendFeed going down? There’s a lot more activity on FriendFeed surrounding my messages (and they aren’t cached in any obvious way) and it’s been down about 1/100th as much as Twitter.

  10. Trevor: well, if every Tweet is copied for every one of its potential readers, that totally explains why Twitter has some scale problems. Most Twitter users don’t use the service very often, if at all (I watch).

    In my scenario there ARE copies. Just not automatic ones. Also, Twitter only needs to keep the last 10 Tweets cached on each user’s page, to keep the home page fast. Other pages take forever to load, so I doubt those are cached. Even in the home page scenario my Tweets would only be copied to those users who haven’t had my Tweets replaced by other users (most of the time my Tweets would be pushed lower, so there wouldn’t be 23,000 copies, only, maybe 1,000).

    Either way, if I’m to blame for Twitter going down, why isn’t FriendFeed going down? There’s a lot more activity on FriendFeed surrounding my messages (and they aren’t cached in any obvious way) and it’s been down about 1/100th as much as Twitter.

  11. I think the charging issue is irrelevant because there are so few of you. If Twitter can’t find a better way to monetize than smacking the <500 Superusers with huge monthly fees they are eventually going to be toast anyway. The Flickr model seems more realistic – charge heavy users a small annual fee and put them on a more robust platform. Heck, I’ll pay just to keep having to hear everybody talk so much about Twitter, the challenges of which seem to have gripped the online community in a dangerously obsessive fashion.

  12. I think the charging issue is irrelevant because there are so few of you. If Twitter can’t find a better way to monetize than smacking the <500 Superusers with huge monthly fees they are eventually going to be toast anyway. The Flickr model seems more realistic – charge heavy users a small annual fee and put them on a more robust platform. Heck, I’ll pay just to keep having to hear everybody talk so much about Twitter, the challenges of which seem to have gripped the online community in a dangerously obsessive fashion.

  13. Scratch the 30 gig, misread Om’s blog. Even so it would be a huge amount of data being moved about Twitter’s internal network given that Scoble is just one of quite a few “super users” on there.

  14. Scratch the 30 gig, misread Om’s blog. Even so it would be a huge amount of data being moved about Twitter’s internal network given that Scoble is just one of quite a few “super users” on there.

  15. I agree with all the commenters saying that Twitter must be copying messages to each follower, but I really hope they’re just copying an ID 25k times and the actual text of the message maybe just a few 10′s of times (i.e. to multiple caches). There’s no way it would perform as well as it does (most of the time) if it were transferring 30 gig of data every time Scoble tweets.

  16. I agree with all the commenters saying that Twitter must be copying messages to each follower, but I really hope they’re just copying an ID 25k times and the actual text of the message maybe just a few 10′s of times (i.e. to multiple caches). There’s no way it would perform as well as it does (most of the time) if it were transferring 30 gig of data every time Scoble tweets.

  17. I don’t believe this, you’re making statements that are 100% false. Yes, every time you tweet, it’s copied 25,000 times. It has to work that way or it wouldn’t have scaled as far as it has. You’re setting yourself up for massive humiliation when you’re definitively proven wrong.

  18. I don’t believe this, you’re making statements that are 100% false. Yes, every time you tweet, it’s copied 25,000 times. It has to work that way or it wouldn’t have scaled as far as it has. You’re setting yourself up for massive humiliation when you’re definitively proven wrong.

  19. Scoble, I think you are fantastic and your enthusiasm for the industry is amazing, but you really should steer clear of the technical arguments.

    You are trying to argue that twitter is using a ‘pivot table’ – so you have one table for users, one table for messages, and a third table that describes your friend relationships. When a query comes into to see a particular users stream you think they ‘mix’ this up, so you do a many-to-many lookup, so for every user (25k in your case) you then look in every one of those users message queues for the most recent messages then mash them together.

    Now they may have started with a ‘obvious’ schema like this about a year ago but I can assure you 1000% that this does NOT scale very far and certainly not up to the point they have got. The reason? because many-to-many lookups in any RDBM are extremely costly and secondly it is very hard to scale across hardware when you build like this, because it is almost impossible to shard because the many-to-many means everyone can potentially be joined together.

    The second methodology described which you laughed at, IS SCALABLE – because you can shard to as many machine as you like for an example lets say each shard owns (10,000) users – each message you send just has to send a tiny signal to each shard of your new message – each shard then looks up within its own local database of 10,000 users to see if any of them are following you. It then adds your message to their queue.

    This is a classic normalization vs de-normalization – you describe normalization in how you think it works – what I hope (and I am sure they are doing a variant of) is de-normalization.

  20. Scoble, I think you are fantastic and your enthusiasm for the industry is amazing, but you really should steer clear of the technical arguments.

    You are trying to argue that twitter is using a ‘pivot table’ – so you have one table for users, one table for messages, and a third table that describes your friend relationships. When a query comes into to see a particular users stream you think they ‘mix’ this up, so you do a many-to-many lookup, so for every user (25k in your case) you then look in every one of those users message queues for the most recent messages then mash them together.

    Now they may have started with a ‘obvious’ schema like this about a year ago but I can assure you 1000% that this does NOT scale very far and certainly not up to the point they have got. The reason? because many-to-many lookups in any RDBM are extremely costly and secondly it is very hard to scale across hardware when you build like this, because it is almost impossible to shard because the many-to-many means everyone can potentially be joined together.

    The second methodology described which you laughed at, IS SCALABLE – because you can shard to as many machine as you like for an example lets say each shard owns (10,000) users – each message you send just has to send a tiny signal to each shard of your new message – each shard then looks up within its own local database of 10,000 users to see if any of them are following you. It then adds your message to their queue.

    This is a classic normalization vs de-normalization – you describe normalization in how you think it works – what I hope (and I am sure they are doing a variant of) is de-normalization.

  21. I think Om’s point was more along the lines that instead of simply bleeding cash, they could make a little. I don’t think he was questioning the architecture.

  22. I think Om’s point was more along the lines that instead of simply bleeding cash, they could make a little. I don’t think he was questioning the architecture.

  23. Robert, I know Twitter isn’t email, why did you bring up Exchange and Bedlam then?

    I’m not sure why you are being so assumptive about their architecture unless some one laid it out to you. Further some of your statements in defense(?) of what they may or may not be doing don’t even make sense.

    I’m not being assumptive. I haven’t said one way or another what they are doing because I have no idea. I only know of the massive large scale systems we have at Microsoft and the relative pros and cons of each. I also know each is designed to meet one general architecturural need and generally these things don’t translate well to serve different kinds of IO. So that you might find is that within any large system you have dozens or more subsystems specifically designed to one scale problem. Some of those will require creating duplicate copies of the data if read performance is required to make your application scale OR be responsive.

  24. Robert, I know Twitter isn’t email, why did you bring up Exchange and Bedlam then?

    I’m not sure why you are being so assumptive about their architecture unless some one laid it out to you. Further some of your statements in defense(?) of what they may or may not be doing don’t even make sense.

    I’m not being assumptive. I haven’t said one way or another what they are doing because I have no idea. I only know of the massive large scale systems we have at Microsoft and the relative pros and cons of each. I also know each is designed to meet one general architecturural need and generally these things don’t translate well to serve different kinds of IO. So that you might find is that within any large system you have dozens or more subsystems specifically designed to one scale problem. Some of those will require creating duplicate copies of the data if read performance is required to make your application scale OR be responsive.

  25. Omar: one problem, Twitter isn’t like email. First of all, only a small percentage of Twitter users ever sign into Twitter again. Let’s say it’s as high as 50% (I think it’s lower). That would mean 12,500 copies. Then, not every one of those users signs in every day. Let’s say only 50% sign in on a particular day. That’s 6,250 copies. But how does Twitter know which users will sign in? It doesn’t. It needs to create the pages on the fly, not copy everything to 23,000 (er, two million) separate tables in a big database. If it did that it’d quickly die, and make it extremely hard to maintain, too.

    Also, many users don’t even use the Web interface. Most of the time I’m looking at messages coming at me in Google Talk. Those are coming one at a time at me. Are you really seriously expecting me to believe that Twitter copies messages 23,000 times before sending them out to me via the XMPP database?

  26. Omar: one problem, Twitter isn’t like email. First of all, only a small percentage of Twitter users ever sign into Twitter again. Let’s say it’s as high as 50% (I think it’s lower). That would mean 12,500 copies. Then, not every one of those users signs in every day. Let’s say only 50% sign in on a particular day. That’s 6,250 copies. But how does Twitter know which users will sign in? It doesn’t. It needs to create the pages on the fly, not copy everything to 23,000 (er, two million) separate tables in a big database. If it did that it’d quickly die, and make it extremely hard to maintain, too.

    Also, many users don’t even use the Web interface. Most of the time I’m looking at messages coming at me in Google Talk. Those are coming one at a time at me. Are you really seriously expecting me to believe that Twitter copies messages 23,000 times before sending them out to me via the XMPP database?

  27. Master: I wasn’t arguing with Dare so much about the architecture. But it’s totally ridiculous to say that my messages are copied 23,000 times. If that were true, then WordPress.com’s architecture would be going down left and right cause this blog would be copied 400,000 times. It is, but in your Web browser, not on the server side.

  28. Master: I wasn’t arguing with Dare so much about the architecture. But it’s totally ridiculous to say that my messages are copied 23,000 times. If that were true, then WordPress.com’s architecture would be going down left and right cause this blog would be copied 400,000 times. It is, but in your Web browser, not on the server side.

  29. Robert,
    It isn’t clear to me why you are taking my post so personally. Regardless of how Twitter is implemented, allowing a user to have 25,000 followers and 25,000 people they are following will cause scale problems. There are different optimizations you could make (Single Instancing is not the panacea you claim, see my post at for http://www.25hoursaday.com/weblog/2008/05/26/SomeThoughtsOnSingleInstanceStorageAndTwitter.aspx more) but it doesn’t change the fact that Twitter has made some bad design and feature decisions.

    As to whether people who generate massive load on the system should be charged…isn’t that a fact of life everywhere else? Internet service providers like Comcast are known to fire customers who use too much bandwidth, in fact your buddy Dave Winer just blogged about that happening to him. Flickr, Y! Mail and a bunch of other services also charge for “pro” features. Why would Twitter pursuing such a business model be so wrong? Would you prefer to have ads in your Twitter streams?

  30. Robert,
    It isn’t clear to me why you are taking my post so personally. Regardless of how Twitter is implemented, allowing a user to have 25,000 followers and 25,000 people they are following will cause scale problems. There are different optimizations you could make (Single Instancing is not the panacea you claim, see my post at for http://www.25hoursaday.com/weblog/2008/05/26/SomeThoughtsOnSingleInstanceStorageAndTwitter.aspx more) but it doesn’t change the fact that Twitter has made some bad design and feature decisions.

    As to whether people who generate massive load on the system should be charged…isn’t that a fact of life everywhere else? Internet service providers like Comcast are known to fire customers who use too much bandwidth, in fact your buddy Dave Winer just blogged about that happening to him. Flickr, Y! Mail and a bunch of other services also charge for “pro” features. Why would Twitter pursuing such a business model be so wrong? Would you prefer to have ads in your Twitter streams?

  31. Now who do I believe when it comes to authority on systems architecture.

    Robert Scoble or Dare Obasanjo?

    lol

  32. Now who do I believe when it comes to authority on systems architecture.

    Robert Scoble or Dare Obasanjo?

    lol

  33. >Open up Twitter… now, did you wait several minutes for your page to appear?

    Open up FriendFeed. Refresh many times. DId the page change? It certainly wasn’t pre-cached before I hit the servers.

    Computers now are fast, if you have the right architecture.

    How does Google work? It is always fast and doesn’t pre-cache all my pages. To do that it’d have to know what I’m thinking before I actually searched for something.

    One thing you haven’t thought about is that even if everything was precached that only a small percentage of my 23,000 followers ever log into Twitter. So, if it’s building a page for each of the 23,000 followers it’s totally wasting resources.

  34. >Open up Twitter… now, did you wait several minutes for your page to appear?

    Open up FriendFeed. Refresh many times. DId the page change? It certainly wasn’t pre-cached before I hit the servers.

    Computers now are fast, if you have the right architecture.

    How does Google work? It is always fast and doesn’t pre-cache all my pages. To do that it’d have to know what I’m thinking before I actually searched for something.

    One thing you haven’t thought about is that even if everything was precached that only a small percentage of my 23,000 followers ever log into Twitter. So, if it’s building a page for each of the 23,000 followers it’s totally wasting resources.

  35. What does Exchange have to do with it? Exchange doesn’t scale (yet) to hundreds of millions of users. Mail systems like Hotmail do in fact keep 25,000 copies of an email if that is how many hotmail users receive the same message. Doing anything else would literally saturate and melt the network (if the architecture was built around a single instance store). There is a physical limit to how much network traffic you can have going between clusters (or scale units). Ideally you optimize your IO for the access patterns of your application (lots of writes and many more reads of unique data).

    BTW, the issue you are describing with Exchange failing is documented here:

    http://msexchangeteam.com/archive/2004/04/08/109626.aspx

    And it wasn’t a failure for the reasons you describe. It has to do with numerous issues and failures unrelated to db scale.

  36. What does Exchange have to do with it? Exchange doesn’t scale (yet) to hundreds of millions of users. Mail systems like Hotmail do in fact keep 25,000 copies of an email if that is how many hotmail users receive the same message. Doing anything else would literally saturate and melt the network (if the architecture was built around a single instance store). There is a physical limit to how much network traffic you can have going between clusters (or scale units). Ideally you optimize your IO for the access patterns of your application (lots of writes and many more reads of unique data).

    BTW, the issue you are describing with Exchange failing is documented here:

    http://msexchangeteam.com/archive/2004/04/08/109626.aspx

    And it wasn’t a failure for the reasons you describe. It has to do with numerous issues and failures unrelated to db scale.

  37. Charging is silly.

    - Money won’t help twitter right now.
    - Charging won’t deter “superusers”.

    They shouldn’t charge, they should ban.

Comments are closed.