Since we created and open sourced Kafka more than 4 years ago, it has seen widespread adoption across thousands of companies like LinkedIn, Netflix, Pinterest, AirBnb, Verizon, Salesforce and more. At Confluent, we are solving a very key problem that companies face, which is collecting data about everything happening inside the company and allowing applications and systems to react to events in real-time.
Apache Kafka is used in different ways, from collecting user activity data, IT/operational metrics to collecting set-top box metrics and stock ticker data, but its biggest differentiator is that it enables collection of large volume data for real time consumption. The impact that this has on the way data flows through an organization, is the ability to deploy a single infrastructure, that can act as the central nervous system transmitting data to the company’s brain, which is all the systems and applications that process data.
Until a few years ago and even now for traditional enterprises, data was locked up in silos. Traditional solutions for handling real time data, like messaging systems, could not keep up with the newer high throughput data sources. And traditional batch ETL tools as well as newer log aggregation systems were not designed for real time consumption. The absence of a single scalable infrastructure for handling variety of data sources, led to an ad-hoc approach to data integration, where people built custom pipes to move data between various sources and destination systems. The problem with this approach is limited data coverage in each of the data processing systems, Hadoop as well as real time systems. Kafka’s popularity can be attributed to the fact that it provides an elegant solution to this age old problem of data movement and it does so reliably in a scalable way.
Web companies are used to managing data in real time, most of which use Kafka to do the same. We see an opportunity in the enterprise who have started to realize this trend and are adopting Kafka.
There is a story behind every company; you can read about ours here. It is truly exciting to get an opportunity to create a company and one that backs the technology you helped create. I look forward to this next phase of my career.