Real-life Meetups Deserve Real-time APIs
First things first, open a linux or mac terminal and tap into the RSVPs stream:
What you’re seeing now is each RSVP to a public Meetup group as we receive it. When a user clicks an RSVP button or an application calls the RSVP method, the server handling the request publishes a message to a RabbitMQ exchange. Streaming API servers then consume a queue bound to this same exchange and publish the message to all of their remote clients.
It sounds simple, and actually, it kind of is. A fanout exchange is perfectly suited to merge all the backend sources of RSVPs into one low-overhead stream, and the engine of this new client facing infrastructure—Netty wrapped in Unfiltered—provides a single interface to write out one message to many clients.
There is one sleight of hand in the implementation. The stream is ephemeral: if you must know what transpired yesterday, don’t ask this service. It has access to all of those RSVPs, of course, but it doesn’t have them in that order. While it could approximately reconstruct the stream as it looked yesterday, it would be contending with web servers for resources that are not optimized for the task. Instead of doing this task poorly, it just doesn’t do it.
That is, it doesn’t provide a full history. It does provide a very recent, precise history of what has gone over the wire. This is implemented as you might expect, with an in-memory bounded queue on each of the stream servers. When API clients disconnect or are disconnected from the stream, they can specify the last
mtime they received; the history will be replayed from that point. There’s no guarantee that the entire history from that instant will be available, but clients that reconnect quickly will not miss a thing.
Far more interesting than this infrastructure, we hope, will be the applications that use it. Whether it’s a live visualization of the data or an indexed analysis of who has been meeting up and where, streaming APIs enable kinds of applications that simply aren’t possible with traditional polling APIs. All without the need for rate limiting or authentication. (Those may prove to be famous last words, but they were fun to type anyway.)
You now have access to the primary unit of Meetup activity—what will you do with it?
Note: It was designed to be easily clustered, but for the moment there’s just one streaming server. Please don’t depend on it for your defibrillator software, etc.