Following is a transcript of a talk that was given at NodeSummit conference in San Francisco, in December 2013.

Part One: Fragmented World.

Good morning. Thank you for coming to such early talk.

I want to share a story with you—a story of how we built an infinitely scalable web API with Node.js. This API allows media organizations to publish and exchange massive amounts of digital content.

When architecting the system, the biggest challenge we faced was that:

We live in an incredibly fragmented world.

When you enjoy your New York Times article online or listen to NPR’s Morning Edition, various pieces of that content come from many different sources. Wire feeds such as Reuters and Associated Press, various social media outlets, partner news organizations–all contribute to the input.

The output is equally fragmented. Once a team of journalists produces the final content, it is pushed to many publishing channels: website, iOS and Android apps, sattelite distribution, as well as: back to the partner organizations and news networks.

Despite the apparent complexity of the process, it has to all work seamlessly. During breaking news or major events coverage, the allowed margin of error is pretty much: zero.

So imagine our excitement when we were asked to build a system that could scale such fully distributed operations for all of public media. The thought of: “Shit, we’re screwed” definitely crossed our minds more than once.

Building a fully distributed system at scale is hard and very risky. We immediately started looking for a success story of something similar. The answer was right in front of us: it’s called the World Wide Web, it’s the most distributed system in the world and we know it works.

Better yet, there’s a lot of research available that explains what makes web architecture scale; including Roy Fielding’s seminal dissertation abour REST architectural style, of course.

Part Two: Hypermedia.

The key to the scalability of web’s architecture is: Hypermedia.

Hypermedia is the matter of which the World Wide Web is made. Much like physical world is built of interacting elementary particles (Bosons and Fermions), the web is essentially the universe of myriad of interacting hypermedia documents.

Despite Hypermedia coming into the spotlight relatively recently, it is no baby. Hypermedia is more or less 23 years old. And much like we ourselves were at that age: it’s largely misunderstood, very rebellious, but also full of potential.

Most of Hypermedia’s potential comes from three core traits:

• the robustness of the HTTP protocol,
• universal use of URLs
• and the extremely versatile media types it uses: HTML and CSS.

Now HTML is really special. It’s just a handful of tags, but the wealth of creative user-interfaces and user experiences that people are able to build with those tags is truly astounding. You have maybe a couple dozen simple rules and you get enormous creativity. With how in the news industry everybody is trying to be original, we knew we needed a media type as versatile as HTML.

Except HTML is mostly optimized for rendering content, and we needed a media type that is instead optimized for describing and publishing of structured content.

The media type we ended up designing is called: Collection Document. Its full specification is documented at the URL: http://cdoc.io. It is based on existing specifications by Hypermedia experts like Mike Amundsen and Mark Nottingham, among others.

Collection Document is a recursive media type that is a document and a collection at the same time. As a document it can describe things. As a collection it contains other documents that can contain other documents etc. This recursion allows us to describe very complex domains.

But it’s also a very simple media type: it has only three top-level elements: attributes, links and errors. Links are basically URLs that describe relationships of various documents as well as: what you can do with the documents: the behaviours.

URLs are absolutely key to Collection Document. The primary key of any document is a URL. The way we describe collections of documents is also through URLs. Permissions, extensibility of semantic meanings of attributes, everything is expressed using URLs. This unified URL-oriented approach allows for some amazing things that I will tell you about a little bit later, but first let me say couple words about how we actually built the a scalable API using this media type and Node.js.

Part Three: The Implementation

NPR is somewhat famous as a champion in the API space. We have long history of building large web APIs. Building APIs, we learned long time ago that for APIs speed is a core feature. If you want people to really use your APIs you need to make them super fast.

This “fast” thing, however, can mean many things depending on a context. In case of a hypermedia API, code spends the majority of its time completing tasks that are I/O bound: most of the time we are storing data into and/or querying it from various services, waiting on input/output over HTTP or various TCP/IP protocols. Node.js was architected to be extremely efficient for such tasks, being non-blocking on I/O. Node was a great fit in that regard.

Node, however is not the only non-I/O-blocking programming environment. But compared to alternatives we really appreciated the community behind Node. Node has a very friendly, strong and healthy community that has produced a wealth of modules. Thanks to these modules, the Node Package Manager (npm) and Node Version Manager (nvm), Node is a very efficient environment for quick development.

There’s also a pragmatic reason for choosing Node. It’s much easier to hire Javascirpt developers or train existing ones on Node than to build a team of, say, Scala experts.

On top of Node.js, we are using Express.js. On top ot Express.js we use Node Bootstrap. Node Bootstrap is this MIT-licensed, open-source project we maintain at http://nodebootstrap.org. It’s not a framework, but rather it’s a skeleton application that captures a lot of best practices of how to organize your project, do clustering and error handling properly, as well as write pluggable modules.

The concept of modules as plugins was essential for us. Operating in a startup-like mode, it was critical for us to be able to prototype quickly. So we designed every aspect of our code to be pluggable: search, storage, queuing system are all swappable.

For instance, search plays a huge role in our implementation. Since we are deployed on Amazon Cloud, we initially used Amazon’s CloudSearch service for search, just because it was readily available. Once we grew out of capabilities of CloudSearch we moved to a very powerful open-source search system: Elastic Search. We had to build our own ElasticSearch cluster at that point, but we didn’t have to rewrite too much of our code, because search implementation was largely pluggable in our system.

This approach of starting small and only hardening swappable components when you have to was monumental for us in being able to run the project in a Lean Startup way.

We ended up open-sourcing a bunch of components from the project at: http://oss.pmp.io. Specifically we were really missing some of the more advanced logging capabilities we were used to in other environments so we open-sourced a framework called metalogger. We also open-sources message-queue based task scheduling framework.

“One More Thing”: Web of APIs

Ok, so this is what we did. We designed some media type, built a Node.js API that we claim is very scalable and open-sourced some projects. What’s the big deal?

Well, it turns out by using Hypermedia architecture, designing the media type with URLs at the center and implementing it all with asynchronous Node.js, we did manage a little more than just ordinary. We ended up with an API that breaks through probably the most significant constraint of modern APIs.

One of the most limiting constraints of the modern APIs is that they are completely silo-ed.

Let’s take one of the most successful APIs we’re all familiar with: Twitter. It’s a landmark one, but it’s also isolated. You can only use the API against the content that is in Twitter’s database. It’s not like you can run a query against a “tweet” that is originated in another system, can you?

But if we think about it—that’s totally anti-web! Elsewhere on the web, you don’t save your page directly into Google’s database to make it searchable! And you don’t have to publish your webpage into somebody else’s database to link to it. You kindof had to do such silly things for the APIs, however.

Well, you don’t have to jail your APIs, anymore. You can set them free. With the API we built you can have core data elements of the API live anywhere on the web, as far as you can reference it! When you use Hypermedia properly and specifically if you use Collection Document media type, you end up with a true web of APIs—A properly interconnected network of APIs the way that web APIs were meant to be!

And u know what? That’s quite amazing. When u can do those kind if things—anything becomes possible!