Data Ingestion Framework
Client
For a US based multinational retail corporation
Business Problem
- Our client wanted to ingest data from multiple RDBMS sources
- They also wanted to develop an ingestion framework to load data into Hadoop environment
Abzooba’s Solution:
- Batch processing framework to ingest the data from different sources to a managed data lake
- ELT framework abstracting the complexity of process on Hadoop platform
- Automic for scheduling and monitoring the orchestrated jobs(workflow)
- Data is transferred from the source using Sqoop connecter for RDBMS, TDCH connector for Teradata and files are processed using Hadoop/HDFS
- Hive with the powerful Tez engine is used for processing the files in HDFS
Business Benefits:
- Improve developer productivity
- Faster and error-free data movement from source to target
- Unified customer view by ingestion of customer and sales data
Tech Stack
