Home>Real-Time Data Ingestion

Real-Time Data Ingestion

Client

For a leading computer memory and data storage device manufacturer

Business Problem

  • Our client wanted to develop a real-time analytics data pipeline
  • They also wanted the capability to monitor and control the pipeline

Abzooba’s Solution:

  • Traffic generation and data ingestion through Kafka directly into Druid DB
  • Real-time analytics cluster hosted as a microservice which reads Kafka IoT events and converts them into insights and metrics
  • A CLI front-end for all components, i.e., traffic generator & analytics cluster
  • Kubernetes for container orchestration
  • Monitoring and log management
  • Microservice framework for traffic generator and data pipeline code
  • System/node level instrumentation for resource usage and performance statistics

Business Benefits:

  • High-speed traffic generation, ingestion, storage and retrieval capabilities​
  • Up to 2 million events/second being ingested through Kafka stream

Tech Stack

Speak to AI expert