Home >
Purpose and Vision
Pain Points
- Non-existence of a single ingestion engine to ingest all types of data
- No homogenous codebase
- Pre-installed cluster needed for the ingestion tool (like Nifi or Sqoop)
- Lack of a cloud-native dynamic ingestion engine
Our Vision
- One code for all the data sources and sink (RDBMS, SFTP, Streaming)
- Spark as the single programming paradigm for all the use-cases
- No pre-installation of cluster needed and minimal resource footprint
- User friendly web interface for Data Source registration, Job config and Job runs
Guiding Principles for Olive
Cloud native design
Platform agnostic (any Cloud/On Premise)
One code serves all (homogenous)
Modular design (plug and play model)
Elastic in nature ( auto scale as needed)
Dynamic compute (provision on-demand)
API driven ( UI, APIs, SDK)
Infrastructure as a code (terraform)
Minimum infra footprint (small VM needed for docker based setup)
Minimal configuration needed on client clusters (minimal Interference)
Benefits of Olive
Agility
- Deploy new use cases in hours…. not days
- Add new data sources with a single click
Flexibility
- Choose any big data platform, on-premise or cloud
- Choose ingestion type like full/incremental, provide analytical queries for ingestion
Cost Efficiency
- Reduce cost and time of building and running analytics use cases
- Reduce cost associated with legacy DW’s
Configurability
- Configurable framework to support any data size
- Cluster setup based on configurations
Operating Efficiency
- Focus resources on generating business value
- Decrease demand on IT through self-service
- Improve reliability in production operation of analytics use cases
Logical view of Olive
Architecture and the Tech Stack
Deployment view
