One of the most important design principles for an API program is to embrace that APIs are products. It follows that successful APIs, much like products, must provide user-optimized experiences. Case in point: it is clearly inappropriate to send a mobile application large response payloads. Your API may get away with it when responding to a request from an enterprise system communicating over a high-speed fiber-optic connection. Some clients may even want to get a rich response, but in general assuming that network is reliable and/or bandwidth is infinite is a fallacy, on the web.

A Brute-Force Approach: Field Filtering

The fact that mobile clients are not fans of large response payloads is hardly a shocking news.

API teams have historically allowed customization of API responses using a somewhat brute-force approach known as “field filtering” – allowing an API client to indicate a list of returned fields in the request. It can look something along the lines of:

  GET https://api.example.com/users/123?fields=firstname,lastname,age

We have also seen solutions where all default fields are allowed, but a client has ability to “ban” potentially expensive fields from the response:

  GET https://api.example.com/users/123?exclude=biography,resume

The extreme case of this approach is: GraphQL, which is basically “SQL” over HTTP, entirely concentrated on querying data-storage on the server and thus tightly coupling clients with the data-models of the server.

There Is a Better Way

One of the main problems with Field Filtering is that it encourages tight coupling of a client with an API server. By allowing such solution, we expect clients to maintain too granular of understanding for your API implementation. This can lead to unnecessary breaking changes and API versioning, which as Mark Nottingham would have you know is evil.

Instead, what we can do is: we can create tiered representations of responses. Each tier of a resource representation can have size-oriented or some other logical name, satisfying the need of tailoring output to a client group, without leading to extraneous coupling. As luck would have it: IETF already has an RFC that suits this purpose: RFC 7240 - Prefer Header for HTTP

This is how you could create “minimal” (vs full “representation”) outputs:

  Get /users/123 HTTP/1.1
  Host: api.example.org
  Content-Type: application/json  
  Prefer: return=minimal
  Vary: Prefer,Accept,Accept-Encoding
  
  HTTP/1.1 200 OK
  Server: nginx/1.4.6 (Ubuntu)
  Date: Sat, 27 Jun 2015 11:03:32 GMT
  Content-Type: application/json; charset=utf-8
  Transfer-Encoding: chunked
  Connection: close
  Vary: Prefer,Accept,Accept-Encoding
  Preference-Applied: return=minimal
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Content-Type

Please note that including “prefer” in the Vary header of the response is required per RFC 7240. It makes sure that some caching infrastructure won’t cache a wrong representation of the resource, since multiple representations will have the same URL, only differentiated by the Prefer header.

You can also see that the response acknowledged the preference by including Preference-Applied: return=minimal in the response.

The RFC only allows two values for the “return” prefer type: minimal and representation. However, it allows defining new preferences. We can use it to define a preference that hints the use-case a client is interested in, and therefore: the fields in the response. Let’s call this preference respond-for.

The name of the preference batch doesn’t always have to be size-oriented. For instance, in case of news media and blogosphere, a logical preference that might make sense, could be:

 Get /blog/1223 HTTP/1.1
 Host: api.example.org
 Content-Type: application/json  
 Prefer: respond-for=teaser
 Vary: Prefer,Accept,Accept-Encoding

We’ve also seen elegant uses of the prefer header when clients ask the API server to transclude some of the data that would normally be just linked from the API response. This can be especially useful if otherwise the client would need to make too many (N+1?) additional requests to retrieve the same data. For instance:

Prefer: transclude=availibility_calendar
Vary: Prefer,Accept,Accept-Encoding

Word of caution: we need to make sure that prefer headers, in our designs, refer to logical batches and not: individual fields. Just moving “Field Filtering” from URL to the HTTP Headers is not a solution, obviously.

Use Business Terms for Prefer

I couldn’t recommend enough the approach of using domain-driven/business terms for batching in the HTTP Prefer approach. The reason being: it leads to batch names that survive changes in your system. As Greg Young (@gregyoung), of CQRS and Event Sourcing fame, says:

Structure changes more often than behavior.

Your use-cases of a system tend to be reasonably stable over long periods of time. How you interpret your internal data: the structure that you use has a tendency of changing a lot. (Source: Code on The Beach 2014)

This has also been my experience in building complex systems, therefore: if you choose business terms for batch names in Prefer, you will be able to defer their change for longer, and as we know: versioning is an evil we avoid, in APIs.

TL;DR - In Conclusion

Do not use “field filtering” to provide more sensibly-sized response payloads to mobile applications or other clients. This may create too tight of coupling between the client and the server. Instead, use HTTP Prefer Header for HTTP - RFC 7240.

Acknowledgements

I first learned this technique (much like most things I know about APIs) from my dear friend and colleague Mike Amundsen. It builds upon thinking and work by amazing IETF contributors: James Snell and Mark Nottingham, as well as probably others in the web community. I apologize if I accidentally missed anybody, but I am very grateful to all who contribute to the future of the better web.