Next Wave of Big Data Will Be Hypermedia
Due to the explosive growth of the volume of the data we operate on, Big Data has been an increasingly hot topic for the past several years. However, on the journey of tackling the Big Data problem, we are still only seeing the tip of an iceberg.
Currently, Big Data is commonly defined as a data-set so large and complex that it becomes impossible to store and process it using traditional, vertically-scalable data management systems. To overcome this complexity, newer systems use horizontal partitioning and parallel processing of data-sets. But the data is still contained and processed in the isolation of a single organization, albeit using new kind of database systems. This will soon change—the end of the siloed Big Data is coming.
The future when Big Data will be more than merely a large and complex data-set is almost here.
Next stage of Big Data is a data-set that crosses the organizational boundaries of a single entity.
We are rapidly approaching the future where the various pieces of a data-set, we need to operate on, are contained in the systems of and owned by different organizations. These pieces will have to be connected over a network.
healthcare.gov is a good example of such distributed system. The main technical challenge of implementing healthcare.gov's backend was that it had to integrate with many existing systems and those systems weren't necessarily ready for such integration.
The $500 million dollar, public fiasco of healthcare.gov is also a vivid indication of just how complex it is to build such systems. Almost the only successful implementation of such large, distributed system is World Wide Web. There's a lot we can learn from the architecture of the web for how to build distributed systems at scale.
Luckily, a lot of effort has already been put in analyzing what in the web's architecture makes it scale. It is: Hypermedia. Therefore it is very sensible to predict that the Big Data systems of the future will be Hypermedia-enabled systems.
I discussed how such hypermedia-enabled Big Data systems may be built in the future in my recent blog post: Web of APIs with Hypermedia and Node.js