Transforming Traditional ETL Processes with Azure

About this project

A healthcare client with heavy reliance on classical Extract, Transform, Load (ETL) services was facing challenges in managing frequent data updates. Employing a team of 10 engineers, they were dedicated to the meticulous task of cleansing, refining, and transforming data to be seamlessly ingested by their enterprise system.

Recognizing the inefficiencies and limitations of this approach, we proposed a shift: migrating the entire data engineering process to Azure Data Factory and related Azure services. This transition was not just about changing platforms; it was about leveraging modern techniques to automate and refine processes that were previously labor-intensive.

Domain & Technology
Our Services Include


CHWE reached out to the Sobah team for this project after failing to get required delivery from 3 different partners in the past.

This problem involves data sources from over 50,000 different sources, where most of the data is unstructured.

Additionally, the user experience of the portal required encouraging citizens to be involved in their politics – a challenge that has eluded younger generations over the last few decades. Most importantly, the initiative is non-partisan and required completely neutral and data driven content to avoid any bias in the insights provided to the citizens.


The client required our team to conduct a thorough and careful study of data sources, architectural choices and innovative search engine optimization techniques. Product involved a sophisticated phased roadmap required to be delivered over a period of years. The core requirement was to build a highly scalable, adaptable and agile product framework and architecture. We were also tasked with engaging content writers to deliver thousands of articles for educating citizens on various topics related with their politics and society. Product envisioned additional roles and content created for analysts and institutions that require bulk data and deeper insights using data analytics and AI.

Our team
The transformative power of the migration was evident almost immediately. Automated processes, combined with the capabilities of Azure, not only accelerated the data engineering but also enhanced its accuracy. One of the most significant outcomes was the dramatic reduction in manpower, dropping the need from 10 full-time engineers to just 3. This optimization, along with the improved accuracy, showcased the tangible benefits of modernizing data processes with Azure.

Got an idea?

Let's participate in the process of creation