Code on Demand in APIs: Dumb Clients, Smart Endpoints
The post was hopefully OK, but much more interesting was the discussion that ensued, in the comments section, with Mike Amundsen about the future role of Code on Demand in APIs. By the end of the debate, we sortof partied at: "it's probably too early for CoD in APIs".
A lot of things have happened since the blog post. Among other things: I got an exceptional privilege of spending last year working right next to Mike and the rest of the amazing team of API experts at the API Academy. I also got much more time to research and think about API design, experiment with various things, since APIs are now my primary job. Independent from me, Hypermedia architectural style has also become significantly more accepted in the industry. We very rarely run into API teams, at this point, who are not either building Hypermedia APIs, have considered it or at least have heard of it. This was definitely not the case year and a half ago.
The need to possibly revisit the question of Code on Demand in Hypermedia, has been brewing in my mind…
Yesterday we had a wonderful API360 Summit in Dallas. Since we were fortunate to have some of the brightest minds as guest speakers and panelists, Summit ended-up into very insightful API design discussions. Furthermore, in the evening Mike, Ronnie, Jason Harmon and I met for drinks where Jason brought the topic of Code on Demand up, again.
While the impressions from the Summit and the following alcohol-enlightened discussion are still fresh in my mind, I want to share with you where I think the status quo of Code on Demand is in APIs now.
Semantic Gap Is Still Very Real
Hypermedia API design is all about bringing the benefits of the universal interfaces, which we enjoy in the Human Web, to the world of the "machine web": the world of APIs. If we can eliminate the spread of wasted effort that stems from each API provider designing essentially the same things in completely incompatible ways and everybody having to write individual clients for each of the thousands of web APIs - we can spend time/talent on more productive work.
But we have a significant problem: Human Web can enjoy a "single" kind of client (a web browser) only because browsers are always operated by humans, and because humans are incredibly skilled in making semantic sense of most unstructured data displayed by web browsers. Machines are virtually incapable of understanding the meaning ("semantics") of anything that is not preprogrammed into them. However "preprogramming" creates tight coupling between an API client and the server, and leads to the waste and breaking changes that we have already mentioned as a significant problem.
The missing ability of machines to understand semantics of the information received is what we commonly refer to as "Semantic Gap" in Hypermedia API design.
There are really only two ways we can address Semantic Gap:
- Make clients "smarter"
- Keep clients "dumb" but send them instructions they need on-demand so that preprogramming (and resulting coupling) is still avoided. Mike Amundsen aptly calls this: "late binding in APIs".
Choice number one is what we have been doing to date. While creating sufficient artificial intelligence is not an immediately viable solution, we can fake "enough" intelligence by limiting and standardizing common semantic vocabularies. Efforts such as: IANA Link Relations repository, microformats and schema.org are all geared towards creating such vocabularies. Once and if we have enough of such vocabularies, we can use them in APIs via profile link relation [RFC6906] and machine-readable profile definition standards such as: ALPS.
While significant amount of work is put in this direction, it is not without challenges. For one: due to human nature, it is quite hard to make people use common anything, let alone vocabularies or standards. And to be fair, it is also hard to reach a point where enough vocabularies are available.
Let's talk about the second choice:
Dumb Clients, Smart Endpoints
A seminal breakthrough in Information Theory and modern communications came from Shannon–Weaver model of communication, which is based on the idea of "dumb pipes and smart endpoints". This model has had vital consequences in everything information-exchange-related, including even things like: Microservices.
I believe that a similar approach of: Dumb Clients, Smart Endpoints can alleviate a lot of challenges of the Semantic Gap. If we can come up with the model where the clients don't need to understand the semantics of communication and can remain largely "dumb", by just understanding processing rules, we can get benefits very similar to those that Shannon and Weaver achieved by keeping transport layer completely agnostic of the information being transmitted.
So, What Are We Waiting For?
I do strongly believe that Code on Demand in APIs will happen. There really is no avoiding it, but I also am very accepting of the idea that timing has to be right.
If we jump into code-on-demand too early, we can hurt ourselves
That said, now that Hypermedia API design is much more widely accepted, now that we have many more people understanding it, I think we can start having intelligent discussions about what Code on Demand is for APIs, where it is appropriate, and how it may look if implemented outside the scope of a traditional web browser.
Is This All "Just Talk"?