Dark Art of APIs in Connected Cars - API Days San Francisco
Today I am going to share with you some of my extremely opinionated thoughts about APIs and the Connected Cars space.
Most of my opinions come from two sources: my more recent API work with The API Academy, and my experience before that— leading tech team at a super-prominent media company in Washington, DC. This seezling hot media company, which should remain anonymous, to the best of my knowledge has highest number of car integrations in the United States. So plenty of battle scars, and fun experiences, with cars there.
I will also share with you some of my strong beliefs that aren’t based on any specific work experience and therefore could be called pure speculation. That is: if you decided to be really mean about it. However, I would much rather you viewed those beliefs as timeless wisdom and accepted them at face value.
Either way, let me share with you my first mind-blowing revelation: There's an all-out, global "arms race" waging in the automotive industry.
As all wars "worth" fighting for go — this one is also primarily about survival and dominance. Car manufacturers, content providers, app developers are all trying to get ahead with the Connected Cars. Leadership in the Connected Cars space is no longer merely a desirable thing. It has become an immediate necessity of very primal and basic nature: one related to survival. Winners of the race will be heftily rewarded, losers may sink in oblivion.
Time is also of great essence:
Conencted Cars space is evolving rapidly. The joint report published by GSMA and Machina Research anticipates that 90 per cent of new cars in 2020 will have some form of in-vehicle connectivity, adding US$600 billion to value of the Connected Life. Current percentage of new cars with connectivity is estimated at less than 10%. Mobile connectivity in cars is predicted to be the top connected application in 2020.
Overall, Connected Car is widely recognized as a game changer for the automotive industry. Experts all agree that selling ‘just’ cars is a thing of the past. Mobility, connectivity and in-car user-experience will be leading decision considerations for car sales.
Unfortunately, alongside enormous opportunities, there're also some significant challenges in the Connected Cars space. The infant state of this market can probably only be compared to the chaos of the smartphones one before iPhone and the App Store.
Last year, Telefonica interviewed most of the major car manufacturers and published a very thoughtful report which summarized 10 principle beliefs that executives at the automotive OEMs hold:
- There is a disconnect between mobile and automotive industry life-cycles
- Connectivity will be necessitated by regulatory mandates (such as the European Commission initiative eCall)
- Car manufacturers are unable to choose between built-in versus brought-in connectivity
- Operators make it a nightmare for OEMs to provide global connectivity (e.g. due to ban on "permanent roaming").
- The Connected Car will cause an upheaval in the traditional dealership model
- Connected Car will lead to new types of ownership models being developed
- Autonomous vehicles are not on the immediate agenda for most OEMs
- Payment models for how consumers pay for Connected Car services are still not yet developed
- Manufacturers remain cautious towards the open app ecosystem
- A connected lifestyle is a given
I am not going to analyze the list, in detail, right now. We unfortunately simply don’t have time for that, in this talk. You can download the original report by Telefonica if you are interested.
The striking thing about the list, however, is: how self-centered it is. It’s predominantly written from OEM’s perspective. It almost exclusively reflects worries, aspirations and interests of car manufacturers first and foremost.
However, Connected Car is inherently a joint ecosystem of: OEMs, aftermarket device manufacturers, content providers and app developers. It’s true that the car is the facilitator, so it may not be the union of the equals, but take away any of the participants and you simply don’t have a sustainable ecosystem, anymore.
To be fair, the desire of automotive OEMs to supremely reign Connected Car environments is understandable—it's a matter of survival, in their eyes, and they are trying to stay in full control. There is but one problem with such approach, and it is most eloquently summarized in the famous quote by Richard Feynman:
"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled"
~ Richard Feynman
or put more simply: our desires can't override the reality.
Let me give you an example of such reality. As I mentioned before, in my previous job I worked for one of the most valuable content providers in the United States that every automotive OEM had significant interest to include in their connected cars. On our side, we also very much wanted to work with them, since connected cars were large potential audience for us. It should have been a highly symbiotic relationship. Yet, work with the OEMs, while rewarding, was also full of barriers and unnecessary pain points.
For each of the integrations, we had to do things completely differently: in one case we were writing a web app, in another: using OEM's binary SDK in our iPhone app to communicate with the car, in yet another one: writing a full in-dashboard experience. And mind you, it is not just the initial development, maintenance of the highly fragmented ecosystem is a much larger headache. Not to mention that our ability to update the apps once they were released was quite limited.
As a content provider and app developer, our developer experience was similar (or even subpar) to what it would have been writing J2ME apps on early, pre-iPhone mobile phones. Nobody can be happy with that experience.
What is worse: because the efforts of integrating with various OEMs are so fragmented, the end-user experience is often also subpar. You do not get slick end-user experience of iPhone apps, you get something much more like J2ME apps. We are pretty sure that is not where user expectations lie, however.
Three Immediate Solutions for Connected Cars
There are three things that automotive industry can do to make things significantly better right away.
Fragmentation is undoubtedly #1 most damaging issue in the Connected Car industry. OEMs all want to do their own thing since they see originality as a way to maintaining differentiation and therefore—competitive advantage. This view is a fallacy. There are way too many car manufacturers for content providers and app developers to keep up with the variety. "Doing your own thing" means that eventually you simply won't have the amount and freshness of apps that your less "controlling" competitors will. OEMs that care about creaitng vibrant app variety in their cars must make app developers' life easier. Wide variety of apps is exactly what users expect in connected cars.
As a solution, some people have suggested that all OEMs should just deploy Android as the base OS. I am not sure it's necessarily the best idea, for couple of reasons:
- Android may objectively not be the best-suited core OS for connected car space. A real-time OS makes significantly more sense for the context. For instance, I would bet that QNX outperforms Android in most car-specific scenarios.
- I have zero confidence that OEMs will be able to all agree on something as fundamental as the core OS. That seems like an unrealistic and unnecessary aspiration. We should shoot for something much more realistic.
In distributed systems we've had much success with the usage of Hypermedia. The most distributed system humankind has ever built — the world wide web — uses a hypermedia type: HTML as its engine. Hypermedia types can serve as the unified communication language for highly non-homogenous networks. This approach has been so effective that some people even tried using HTML5 in the car space. I see using HTML5 as a stretch, since HTML5 is a hypermedia format created for a different context and not optimized for cars, but the intent seems right.
I believe the silver lining to be in creating a Connected Car-optimized hypermedia format that can energize the space just like HTML did for the World Wide Web. There's already a large body of work in custom hypermedia types. I believe Connected Cars hypermedia type can be based on an existing popular, generic type, such as: Uber, HAL, Siren or Collection.json.
A good example of taking a generic hypermedia type and customizing it for industry-specific needs is: Collection.Document, which was based on Collection.json and created for news media industry.
There's enormous anxiety related to security in the Connected Cars. People are terrified by a thought of cars getting "hacked". This may be a bigger issue for "self-driving" cars, but the solution for Connected Cars lies in:
- Compartmentalization — core car functions should be highly guarded. No third-party app should ever get any access to core driving functions like handling and/or braking.
- Using standard, proven, delegation-based and contextual security framework such as OAuth that allows granular access to specific system features.
We are already very familiar with the second one: when I install an app on my mobile phone it doesn't just get access to everything. My phone asks whether the app should get access to: GPS, contacts, internet etc. We have very granular control over deciding if app is doing what we assume it should be doing or want it to be doing. Automotive apps need same kind of granular access.
Currently, only the lucky few can get access to OEMs and get their apps in a Connected Car. Almost the only way to get your app in, is to have partnerships with the car manufacturers. In most cases you can't even get access to documentation without a group of lawyers signing stacks of papers.
There's absolutely no way to develop Connected Car space by keeping it as a tightly-held, closed system. On the contrary, OEMs should build developer communities by providing things that communities require: documentation, self-service portal, sandboxes, kits, marketplace etc.
Multiple Ways to View the World
This is where I could have ended my talk—by outlining three immediate steps that can be taken to improve Connected Cars space significantly.
But I'm really passionate about this space and I feel like we often talk about specifics, without stepping back to analyze the big picture. To paraphrase the "give fish vs. teach fishing" saying: we often talk about how to fry the fish, and don't spend enough time on figuring out how fish is caught in the first place.
Connected Cars is a special case of Internet of Things (IoT). The context of IoT, forces driving it, are different enough that it requires fundamentally different kind of thinking, different approach to system design/architecture.
The challenges facing the Connected Cars are significant enough that we are going to need little bit of help from some great minds. Specifically the following four:
Loosely, in the order or decreasing amount of hair, we have: Isaac Newton, Albert Einstein, Niels Bohr and Roy Fielding.
First three are important because they all had seminal impact on humankind's understanding of how physical world works. Roy Fielding, I would argue, had similarly seminal impact on our understanding of how Internet, the world wide web specifically, works and scales.
Scale, in general, has a lot to do with what model of world-view is appropriate for a problem space.
Isaac Newton had very mechanistic, highly organized view of the world: actions have reactions, if we know all participants and all rules, we can predict the future based on solid physical rules. This view of the world is the most comfortable for us, humans, since it brings the comfort of order and predictability. As you may remember from high-school physics, Newton's model works reasonably well in highly controlled, average-distance (scale) scenarios. In networked environments, I would compare this scale to that of enterprise systems: integration of units of a single organization or data-systems of partner organizations. Such environment is highly controlled, usually interactions are limited to within a single data-center or on excellent connectivity, behind trust boundaries, with high level of predictability.
Einstein turned Newton's world upside down. He asserted that at large scale everything is relative: time is relative, distance is relative, signals don't propagate instantaneously, speed of light is the highest theoretical speed possible. Many things we had held sacred and reliable, time most importantly, became relative for us. I would say, World Wide Web and Web APIs are a good analogy of Einstein's world: distances are long enough that many of the relativisms Einstein preached kick in effect. We cannot, anymore, rely on instantaneous execution at large distances. Since time is relative, locks at distance are unreliable and hence transaction-management is often impossible.
The complications introduced by "web scale" are a good explanation to why old SOA architectures did not scale on the web. Web APIs aren't just web services thrown at mobile applications. They are fundamentally different in having to work at large distances. We learned a lot about how to do Web APIs right thanks to iPhones and Androids, using RESTful and Hypermedia APIs.
The story doesn't end there, however. Somebody else turned upside down Einstein's world as well. It was Niels Bohr and Quantum Mechanics. Bohr said that not only mechanistic laws fall apart at large distances, but Einstein's enhancements to Newton's formulas are insufficient when we want to operate at very small, atomic and molecular distances. At these distances, not only things are relative, but they are inherently probabilistic. You can never precisely measure anything, since the act of measurement alters the value being measured.
Ironically, the thought of inherently probabilistic values was so radical back in the day that even Einstein, brilliant mind that he was, never could accept such notion of randomness. To his death Einstein rejected Quantum theories and even famously declared that: "God doesn't play dice". Of course, since then Quantum Mechanics has been proven by series of experiments and its applicability has been widely established.
I believe that Quantum Mechanics is analogous to IoT in that both operate at scales with high density of things. If on the web we had hundreds of millions or handful of billions of connected devices, with IoT that number is expected to grow hundred or thousand times. The rules and laws that we used for the Web won't necessarily be applicable with such dense space. Just like Einstein's world-view had to be augmented by Bohr, we will see fundamentally different approaches to tackle IoT space.
At very small distances, in low-energy and small devices environments we need more peer-to-peer solutions, probabilistic, Fuzzy logic and architectures that work well in that context.
I believe that Connected Cars space will combine web technologies with IoT technologies. In-the-car communication may well be predominantly Bluetooth-based while car-to-cloud communication will use familiar APIs, possibly enhanced to withstand patchy connectivity.
Most Importantly: It Has to Be Fun.
Connected Cars space is in its very early infancy. We will learn much more about how to do it right. As we learn, we must have fun, however.
Engineering is the field where excitement is crucial for achieving great results. We all got started in this "racket" because it was enormous fun to tinker with machines and make computers do things. That early excitement has to last if we are to create great products.
In the example I mentioned before, when we had to integrate with various car systems in various inefficient ways — yes it was unproductive, but most importantly it was wrong simply because: it wasn't fun.