Home>Data Ingestion Framework

Data Ingestion Framework

Client

For a US based multinational retail corporation

Business Problem

  • Our client wanted to ingest data from multiple RDBMS sources ​
  • They also wanted to develop an ingestion framework to load data into Hadoop environment

Abzooba’s Solution:

  • Batch processing framework to ingest the data from different sources to a managed data lake​
  • ELT framework abstracting the complexity of process on Hadoop platform​
  • Automic for scheduling and monitoring the orchestrated jobs(workflow)​
  • Data is transferred from the source using Sqoop connecter for RDBMS, TDCH connector for Teradata and files are processed using Hadoop/HDFS
  • Hive with the powerful Tez engine is used for processing the files in HDFS

Business Benefits:

  • Improve developer productivity
  • Faster and error-free data movement from source to target
  • Unified customer view by ingestion of customer and sales data

Tech Stack

Speak to AI expert