About

This project presents the design and implementation of an end-to-end Lakehouse solution using Microsoft Fabric. The objective was to modernize analytics by moving data from a transactional Azure SQL database and flat files into a scalable Bronze, Silver, and Gold architecture. The solution enables faster reporting, simplified data modeling, and reliable insights for business users through automated pipelines and a clean star schema model. Enabled faster and more consistent business reporting by providing a single, trusted analytics layer for sales insights.

To manage a large one-time data load in a controlled and scalable way, a control table is used to drive ingestion logic. This solution performs a one-time data load from a large Azure SQL database into the Fabric Bronze layer(Lakehouse). Since the data volume is high, the load is split into smaller batches instead of loading everything at once. A control table is used to define batch ranges and source queries, enabling dynamic ingestion without hardcoding logic in the pipeline. This approach allows the pipeline to automatically loop through batches and ensures a scalable and controlled load process

​To handle the incoming daily data volume efficiently, incremental ingestion is implemented after the initial batch load. The pipeline first reads the last processed watermark value and then ingests only new or updated records into the Bronze layer. Once the load completes, the watermark is updated to ensure subsequent runs process only fresh data. This approach minimizes data movement, improves performance, and supports reliable daily data refreshes.