With fragmented IT infrastructure, businesses are sitting on more than 1 separate databasesto create individual reports. Consolidation of report on-demand still remains a challenge.
Helped migrate 3 existing SQL server database on Azure Data lake. Apache Nifi was used for Data Logistics Platform and interactive dashboards was created with Tableau and Talend as the ETL/ELT tool. Apache Hive was used to create the BI Pipeline for Structured Data and Kafka was used for unstructured data like EMR Notes. The infrastructure has the capabilities to ingest EMR Notes and scanned prescriptions analysis.
Apart from on demand reporting, a separate pipeline was created using handcrafted R/Python algorithms to implement the Predictive Analytics Use Cases like Forecasting Patient Inflow. The existing logic from SRSS reports were migrated into a Business Layer under 4 data zones with separate Metadata and Business layers. Finally, it was imperative to have Governamce, lineage and Quality layers in the formation of Data Lake. Single Sign On and Authentication provided through Active Directory was made HIPAA/MARE-E/HITRUST compliant with detailed Healthcare PII security audits in place.
Lowering of projected infrastructural expenses by one-third compared to a DW Solution and Healthcare Data Lake from desperate sources.
The solution helped in ingesting unstructured data in an efficient manner with the analytics workbench used for creation of new reports and predictive models for instant publishing.