RSS shows its age in real-time web (SUP and XMPP to the rescue?)

The real time web is coming at us very quickly, but it exposes major problems in our RSS/Atom infrastructure.

What is the real-time web?

You can get a small taste of that by watching the 5,300+ people I’m watching in Real Time on friendfeed.

The first time I saw the real-time web, I saw it when my tweets showed up on Twitter search and friendfeed within minutes. Sometimes within seconds. Now, imagine a world where everything worked like that. That’s the real-time web.

The problem is that our blogs don’t participate in the real-time web. They publish via RSS. RSS is not real time. RSS only publishes when a service like Google Reader asks for it. It has no way to wave its hand and tell your reader “hey, there’s something new here for you to get.” So, most RSS aggregators just visit on a regular basis, looking every few minutes to see whether something new has shown up.

For blogs that’s just fine. After all, most blogs take a few minutes to a few hours to write and it won’t kill you if you don’t read my words here for 20 minutes or longer.

But there’s a new expectation that we’re having thanks to Twitter. We want everything now in real time. I want to see everything that was published now and respond to it now and I want to have conversations about all that in real time.

This works on Twitter and friendfeed, which were built on real-time principles (er, messaging principles) rather than Web principles.

But when you try to hook the real-time web up to the old creaky RSS web, well, you see that the two aren’t very compatible.

Today I tried to setup an ego feed where I could track stuff that uses my name from around the web in real time. It doesn’t work very well. It’s slow. And, worse, friendfeed can’t tell where the original item came from so it gives it a generic RSS icon. So, it’s not only not real time but it’s ugly as well. I talked more about that with a bunch of people on friendfeed today.

So, what’s the answer?

Well, the geeks are exploring two technologies.

The first is XMPP. This is protocol developed for instant messaging applications but Twitter and friendfeed and others have adopted it. This is why when you Tweet the message shows up in friendfeed so fast.

The second is SUP. This was designed by friendfeed to be more efficient, like RSS. But with the added benefit that the feed provider can raise its hand and say “I have something new forĀ  you.” This makes real-time feeding possible, as developer Jeff Smith demonstrates when he built a system that shoved data into friendfeed in just a microsecond.

The third is GNIP, which is trying to build a service that stands between all sorts of services that are supporting the real-time web.

The problem? Very few services that could help the real-time web evolve are using either of these two protocols.

In fact, I was shown a real-time news service that’ll come out in March that didn’t use either of these protocols. Why? They didn’t even know that a real-time web was evolving on Twitter and FriendFeed and that there are dozens of tools like Twhirl and TweetDeck that are built on top of those too. Which is why I’m writing this post.

If you’re a developer, are you thinking about how to make your feeds real time? Why not?

One reason I can see is that it increases the bandwidth needed, especially if you’re pushing out a lot of data. So, in this harsh economic times developers might be unwilling to spend more resources. But there are some things, like searches, that need real-time results. I’d love to hear what developers are thinking here about balancing the need for low-cost systems with real-time publishing.

More info on SUP and the real-time web:

Paul Bucheit, co-founder of friendfeed, started a whole discussion about it.
OurDoings, a photosharing service, was one of the first services that supported the real-time web on friendfeed and they wrote about their experiences with SUP here.
The friendfeed blog has more info on the release of SUP.
Derek van Vilet made a WordPress plugin for SUP and explains that here.

UPDATE: Mike Taylor says I should have mentioned some XMPP resources in this friendfeed conversation. Here’s the ones he recommended:

Jonathan Jesse, in same friendfeed thread, added: “Robert: on Leo Laporte’s FLOSS Weekly they covered XMPP with one of the developers and the guy who writes the documentation for Jabber is a great overview of XMPP and more info: ”