Microsoft Fabric Overview

Microsoft Fabric is a unified, end-to-end analytics platform introduced in May 2023 that integrates multiple data and analytics services into a single software-as-a-service (SaaS) experience. It is built on a common foundation called OneLake, a single, multi-cloud data lake that serves as the single source of truth for all data within the organization. Fabric eliminates the need to stitch together separate services like Azure Data Lake Storage, Azure Synapse Analytics, Azure Data Factory, and Power BI, which previously required complex integration and separate management.

Before Fabric, organizations typically built analytics solutions by combining multiple Azure services: Azure Data Lake Storage for data storage, Azure Data Factory for data ingestion, Azure Synapse Analytics for data warehousing, Azure Databricks for data engineering and data science, and Power BI for visualization. This approach led to several challenges: - Data silos: Each service had its own storage, making it difficult to share data across teams. - Complexity: Managing multiple services with different authentication, governance, and monitoring. - Cost: Duplicate storage and compute resources. - Latency: Moving data between services introduced delays.

Fabric addresses these by providing a unified platform where all workloads share the same underlying storage (OneLake), a common security and governance model, and a consistent user experience across data engineering, data science, data warehousing, real-time analytics, and business intelligence.

Fabric is built on a foundation of OneLake, which is a single, logically centralized data lake that automatically partitions and replicates data across regions. OneLake is built on top of Azure Data Lake Storage (ADLS) Gen2 but abstracts the underlying storage details. Every Fabric tenant gets a single OneLake that is automatically provisioned. Within OneLake, data is organized into workspaces (similar to containers) and items (the different artifacts like lakehouses, warehouses, reports).

Key architectural components: - OneLake: The data lake that stores all data in open formats (Delta Parquet, CSV, etc.). It supports shortcuts, which are symbolic links to external data sources (e.g., Azure Data Lake Storage, Amazon S3) without moving data. - Compute engines: Fabric provides multiple compute engines that can read from OneLake directly: - Spark (for data engineering and data science) - SQL (for data warehousing) - Power BI (for analytics and reporting) - Real-Time Analytics (Kusto Query Language) - Data Factory: A cloud-based ETL/ELT service that orchestrates data movement and transformation. - Semantic model: A business-level abstraction layer that defines metrics, relationships, and calculations for Power BI.

Fabric includes the following workloads (also called experiences):

Default values and timers:

To create a Fabric workspace: 1. Navigate to the Fabric portal (app.powerbi.com or fabric.microsoft.com). 2. Click on "Workspaces" and select "New workspace." 3. Provide a name, description, and set access permissions. 4. Once created, you can add items like Lakehouse, Data Warehouse, etc.

To verify OneLake integration:

Fabric integrates with: - Azure Active Directory: For authentication and authorization. - Azure Purview (Microsoft Purview): For data cataloging and governance. - Azure DevOps: For CI/CD pipelines. - Git: For version control of notebooks and reports. - Microsoft 365: For embedding reports in Teams and SharePoint. - Dynamics 365: For accessing business data.

Fabric also supports open standards like Delta Lake, Parquet, and T-SQL, making it compatible with third-party tools that support these formats.

For DP-900, you need to know:

A large retail chain with 500 stores uses Microsoft Fabric to unify sales data from point-of-sale systems, inventory data from warehouses, and customer data from CRM. Previously, they used separate Azure Data Lake Storage for raw data, Azure Databricks for processing, and Power BI for reporting. Data movement between services caused delays and inconsistencies. With Fabric, they create a single Lakehouse that stores all data. Data Factory pipelines ingest nightly sales data from each store's SQL Server database into the Lakehouse. Spark notebooks clean and aggregate the data. The Data Warehouse workload provides fast SQL queries for inventory analysts. Power BI reports are built directly on the Lakehouse, showing real-time inventory levels and sales trends. OneLake shortcuts allow referencing supplier data stored in an external S3 bucket without copying. The company reduced data latency from 24 hours to under 1 hour and cut storage costs by 40% by eliminating duplicate copies.

A healthcare provider aggregates patient data from multiple electronic health record (EHR) systems, lab results, and wearable devices. They need to comply with HIPAA and ensure data governance. Fabric provides a unified platform where all data resides in OneLake with common security policies. Real-Time Analytics workload ingests streaming heart rate data from wearables using Kusto Query Language. Data Science workload builds predictive models for patient readmission risks. Data Warehouse stores structured patient records. All workloads share the same data, so a model trained in Data Science can be deployed to score new data in Real-Time Analytics. The organization uses Microsoft Purview for data cataloging and lineage. Common misconfiguration: forgetting to set appropriate row-level security on the Lakehouse, leading to unauthorized data access. Best practice: use workspace roles and item permissions to enforce least privilege.

A global bank uses Fabric for risk analysis and regulatory reporting. They have petabytes of transaction data in on-premises Hadoop clusters. With Fabric, they create shortcuts to the on-premises data using OneLake's ability to connect to ADLS Gen2 and then to on-premises via Azure Stack HCI. Data Factory pipelines copy critical data into the Lakehouse. Data Engineering workload runs daily risk calculations using Spark. The Data Warehouse workload provides low-latency queries for compliance officers. Power BI reports are embedded in a custom web application for real-time risk dashboards. Performance consideration: Spark pool configuration must be tuned for the large data volumes; default settings may cause out-of-memory errors. Misconfiguration: not partitioning large tables, leading to slow queries. Fabric's auto-partitioning helps but explicit partitioning by date is recommended.