Home>Data Ingestion & Transformation

Data Ingestion & Transformation

Client

Global retail conglomerate’s US arm 

Business Problem

  • Our client wanted to fetch data from existing multivariate sources, perform transformations, and write it back to destination storage​
  • Orchestrating and designing data pipelines

Abzooba’s Solution:

  • Data Sources – Mainframe files (VSAM, SEQ), DB2, Informix DB, SQL Server & Oracle
  • ETL from data source using Azure Databricks
  • Databricks & HDInsights on Azure Cloud
  • Storage of source data, intermediate and destination data in ADLS Gen2
  • Creation of data pipelines in Azure Data Factory(ADF)
  • Failure e-mail notification using Logic Apps
  • Azure DevOps as code repository and for versioning
  • HDFS using Hive for storing source tables, testing, and debugging

Business Benefits:

  • Output files(CSV) on runtime (either once or multiple times a day)

Tech Stack

Speak to AI expert