Deepstream: an Open-source Server for Building Realtime Apps

Realtime apps are getting really popular, but they’re also hard to build. Wolfram Hempel introduces deepstream, an open-source server he co-founded to make data-sync, request-response and publish-subscribe a whole lot easier. Realtime is eating the world! Or at least it’s taking bigger and bigger bites. Whether it’s collaborative editing in Google Docs, chatting via Facebook messenger, financial trading on the move, IoT controls, live dashboards or multiplayer gaming — users are increasingly expecting to see changes happen as they happen.

Even traditionally static sites like social networks or forums are starting to abandon the refresh button and instead stream updates directly into your feed. But as popular as realtime apps are, they’re also hard to build. While it’s possible for smaller projects or POCs to introduce realtime features just by adding a pinch of Socket.io, large-scale use cases require a fundamentally different architecture. Concepts like concurrent connections, failover, streaming data-consistency, persistence, encryption and permissioning all have to be woven into the fabric that powers this new generation of apps.

One industry where I learned this only too well is investment banking. The servers that power the myriad of flashing screens on the world’s trading floors are monolithic beasts, complicated and breathtakingly expensive. But they are fast. Very fast. And they’ve got something else right: they use a concept called “data-sync”.If you’ve already built a realtime app, chances are you’ve used a pattern called “publish-subscribe” or “pub-sub” for short: subscribers listen for events on a channel and others publish these events...