Data Engineering with Microsoft Fabric

Introduction into Microsoft Fabric

The chapter introduces the data lake approach. It also provides a high-level overview of the building blocks of Microsoft Fabric and how to get started. The Data Mesh architecture is discussed and compared with Microsoft Fabric.

• What is Microsoft Fabric?
• From traditional data warehousing to data lakes
• Data Mesh Architecture
• Personas in Microsoft Fabric
• Working with Task Flow
• Microsoft Fabric Licensing
• Monitor Microsoft Fabric
• Domains and Workspaces in Microsoft Fabric
• LAB: Getting started with Microsoft Fabric

Storing Data in OneLake

OneLake provides a single, unified, logical data lake for your whole organization. Like OneDrive, OneLake comes automatically with every Microsoft Fabric tenant and is designed to be the single place for all your analytics data.

• Common Big Data storage formats
• OneLake and Lakehouses in Fabric
• Source Control integration
• Setting up deployment pipelines
• Creating Lakehouses
• Working with files and folders in a Lakehouse
• Implementing a medallion architecture using Lakehouses
• Securing data in OneLake
• Working with Shortcuts
• LAB: Setting up Lakehouses in OneLake

Working with Delta Tables

Delta Lake is an optimized storage layer that provides the foundation for storing data and tables in a Fabric lakehouse. Learn how to create, query and optimize Delta Tables in a Microsoft Fabric.

• what is a Delta Lake
• Working with Delta Tables
• Managing Schema change
• Version and Optimize Delta Tables
• LAB: Working with Delta Tables

Getting started with Data Factory

Data Factory allows you to ingest, prepare and transform data from a rich set of data sources like databases, files, cloud data sources,... This chapter illustrates how to use Activities to build pipelines that ingest data in a Lakehouse.

• What is Data Factory ?
• Pipelines vs Dataflows in Data Factory
• Copy Data into a Lakehouse using Pipelines
• Adding activities to a Pipeline
• Working with precedence constraints
• LAB: Ingesting data using Pipelines

Authoring advanced Pipelines

This module dives deeper into the process of building an Fabric pipeline. The module mainly focusses on how to work with expressions, variables and parameters to make dynamic pipelines.

• Working with Expressions
• Variables and Parameters
• Using Looping and Conditional Logic in pipelines
• Debugging a pipeline
• LAB: Authoring and debugging advanced Pipeline

Ingest and Transforming data using Dataflows

With Dataflows you can visually design data transformations without the need to learn yet another tool or language. Dataflows in Microsoft Fabric are based on Power Query Online.

• Creating Queries to load data
• Applying Transformations
• Appending and Merging Queries
• Query Folding
• Using Dataflows inside a Pipeline
• Managing connections
• LAB: Ingesting and transforming using Dataflows

Building a Synapse Data Warehouse

A Synapse Data Warehouse is a database that stores data in OneLake and provides a medium to interact with the database using SQL commands.

• The SQL analytics endpoint of the Lakehouse
• Creating tables in a Synapse Data Warehouse
• Ingesting data using pipelines
• Ingesting data using T-SQL
• Querying the Warehouse
• The Default Power BI semantic model
• LAB: Creating and using a Warehouse

Synapse Data Engineering using Spark

Data engineering is the process of designing and building systems that let people collect and analyze raw data from multiple sources and formats. Using popular languages such as Python, SQL and R data can be loaded, transformed and analyzed via interactive notebooks.

• Introducing Apache Spark
• Creating Environments or Apache Spark clusters
• Working with Notebooks in Fabric
• Magic commands
• Visual Studio Code integration
• Scheduling Notebooks
• Microsoft Fabric decision guide: Copy activity, Dataflow or Spark
• LAB: Getting started with Notebooks in Microsoft Fabric

Data wrangling using PySpark and Spark SQL

PySpark and Spark SQL allow users to perform complex data processing tasks with few lines of code using Notebooks.

• The SparkSession, SparkContext and SQLContext objects
• Reading and writing data using DataFrames
• Data Cleansing using PySpark
• Grouping and aggregating data in PySpark
• Joining DataFrames
• Using Spark SQL to select and manipulate data
• Visualizing data using Notebooks and DataFrames
• LAB: Data wrangling using PySpark and Spark SQL

Synapse Real-Time Analytics in Fabric

Real-Time Analytics is a fully managed big data analytics platform optimized for streaming, time-series data. It contains a dedicated query language and engine with for searching structured, semi-structured, and unstructured data in close to real-time.

• Creating a KQL database
• Ingesting data into tables
• Query data using KQL
• Create and manage EventStreams
• LAB: Working with Real-Time Analytics

Reporting in Fabric

Power BI transforms your company's data into rich visuals for you to monitor your business and get answers quickly. Learn how to connect to your data stored in Microsoft Fabric using Power BI.

• Creating Power BI Reports
• DirectQuery vs Import with Microsoft OneLake
• Using and configuring Direct Lake mode
• LAB: Creating Power BI Reports

Data Activator

Data Activator in Microsoft Fabric takes action based on what's happening in your data. Learn how to setup conditions against your data and trigger actions like run a Power Automate Flow when the conditions are met.

• Creating and using Reflexes
• Defining Triggers, Conditions and Actions
• Getting data from Reports or EventStreams
• LAB: Use Data Activator in Fabric