Creating Client-Optimized Resource Representations in APIs
One of the most important design principles for an API program is to embrace that APIs are products. It follows that successful APIs, much like products, must provide user-optimized experiences. Case in point: it is clearly inappropriate to send a mobile application large response payloads. Your API may get away with it when responding to a request from an enterprise system communicating over a high-speed fiber-optic connection. Some clients may even want to get a rich response, but in general assuming that network is reliable and/or bandwidth is infinite is a fallacy, on the web.
A Brute-Force Approach: Field Filtering
The fact that mobile clients are not fans of large response payloads is hardly a shocking news.
API teams have historically allowed customization of API responses using a somewhat brute-force approach known as “field filtering” – allowing an API client to indicate a list of returned fields in the request. It can look something along the lines of:
We have also seen solutions where all default fields are allowed, but a client has ability to “ban” potentially expensive fields from the response:
The extreme case of this approach is: GraphQL, which is basically “SQL” over HTTP, entirely concentrated on querying data-storage on the server and thus tightly coupling clients with the data-models of the server.
There Is a Better Way
One of the main problems with Field Filtering is that it encourages tight coupling of a client with an API server. By allowing such solution, we expect clients to maintain too granular of understanding for your API implementation. This can lead to unnecessary breaking changes and API versioning, which as Mark Nottingham would have you know is evil.
Instead, what we can do is: we can create tiered representations of responses. Each tier of a resource representation can have size-oriented or some other logical name, satisfying the need of tailoring output to a client group, without leading to extraneous coupling. As luck would have it: IETF already has an RFC that suits this purpose: RFC 7240 - Prefer Header for HTTP
This is how you could create “minimal” (vs full “representation”) outputs:
Get /users/123 HTTP/1.1 Host: api.example.org Content-Type: application/json Prefer: return=minimal Vary: Prefer,Accept,Accept-Encoding HTTP/1.1 200 OK Server: nginx/1.4.6 (Ubuntu) Date: Sat, 27 Jun 2015 11:03:32 GMT Content-Type: application/json; charset=utf-8 Transfer-Encoding: chunked Connection: close Vary: Prefer,Accept,Accept-Encoding Preference-Applied: return=minimal Access-Control-Allow-Origin: * Access-Control-Allow-Headers: Content-Type
Please note that including “prefer” in the
Vary header of the response is
required per RFC 7240. It makes sure that some caching infrastructure won’t
cache a wrong representation of the resource, since multiple representations
will have the same URL, only differentiated by the
You can also see that the response acknowledged the preference by including
Preference-Applied: return=minimal in the response.
The RFC only allows two values for the “return” prefer type:
representation. However, it allows defining new preferences. We can use it
to define a preference that hints the use-case a client is interested in, and
therefore: the fields in the response. Let’s call this preference
The name of the preference batch doesn’t always have to be size-oriented. For instance, in case of news media and blogosphere, a logical preference that might make sense, could be:
Get /blog/1223 HTTP/1.1 Host: api.example.org Content-Type: application/json Prefer: respond-for=teaser Vary: Prefer,Accept,Accept-Encoding
We’ve also seen elegant uses of the
prefer header when clients ask the API
server to transclude some of the data that would normally be just linked from
the API response. This can be especially useful if otherwise the client would
need to make too many (N+1?) additional requests to retrieve the same data. For
Prefer: transclude=availibility_calendar Vary: Prefer,Accept,Accept-Encoding
Word of caution: we need to make sure that prefer headers, in our designs, refer to logical batches and not: individual fields. Just moving “Field Filtering” from URL to the HTTP Headers is not a solution, obviously.
Use Business Terms for Prefer
I couldn’t recommend enough the approach of using domain-driven/business terms for batching in the HTTP Prefer approach. The reason being: it leads to batch names that survive changes in your system. As Greg Young (@gregyoung), of CQRS and Event Sourcing fame, says:
Structure changes more often than behavior.
Your use-cases of a system tend to be reasonably stable over long periods of time. How you interpret your internal data: the structure that you use has a tendency of changing a lot. (Source: Code on The Beach 2014)
This has also been my experience in building complex systems, therefore: if you choose business terms for batch names in
Prefer, you will be able to defer their change for longer, and as we know: versioning is an evil we avoid, in APIs.
TL;DR - In Conclusion
Do not use “field filtering” to provide more sensibly-sized response payloads to mobile applications or other clients. This may create too tight of coupling between the client and the server. Instead, use HTTP Prefer Header for HTTP - RFC 7240.
I first learned this technique (much like most things I know about APIs) from my dear friend and colleague Mike Amundsen. It builds upon thinking and work by amazing IETF contributors: James Snell and Mark Nottingham, as well as probably others in the web community. I apologize if I accidentally missed anybody, but I am very grateful to all who contribute to the future of the better web.