Course Overview
The Microsoft Azure Databricks Data Engineer Associate (DP-750) course is designed for data engineers and analytics professionals who build scalable data engineering solutions using Azure Databricks and Apache Spark on Microsoft Azure. This instructor-led training provides comprehensive coverage of designing, implementing, securing, and optimizing modern data pipelines using the Databricks Lakehouse Platform.
Participants will learn how to configure Azure Databricks workspaces, manage compute resources, develop notebooks, ingest data from multiple sources, transform large datasets, and build reliable batch and streaming pipelines. The course covers core data engineering technologies including Delta Lake, Auto Loader, Structured Streaming, Unity Catalog, SQL Warehouses, and Databricks Workflows to support enterprise-scale analytics solutions.
Learners will gain practical experience implementing the Medallion Architecture (Bronze, Silver, and Gold layers), managing Delta Lake tables, optimizing Spark workloads, implementing enterprise data governance, and securing data assets using Microsoft Entra ID integration, Unity Catalog, Azure Key Vault, and role-based access control. The course also introduces Lakeflow Jobs for workflow orchestration and explores performance optimization techniques using Photon, Adaptive Query Execution (AQE), partitioning, and caching.
Extensive hands-on labs and real-world implementation scenarios enable participants to build end-to-end data engineering solutions using Azure Databricks. By the end of the course, learners will be equipped with the practical skills required to design secure, scalable, and governed Lakehouse solutions and prepare confidently for the Microsoft Certified: Azure Databricks Data Engineer Associate (DP-750) certification.
Course Objective
Upon successful completion of this course, participants will be able to:
- Understand the architecture and core components of Azure Databricks and the Databricks Lakehouse Platform.
- Configure Azure Databricks workspaces, compute resources, SQL Warehouses, and development environments.
- Develop collaborative notebooks using Python, SQL, Spark SQL, and Apache Spark.
- Ingest structured, semi-structured, and streaming data using Azure Data Lake Storage, Auto Loader, and Apache Spark.
- Build scalable ETL and ELT pipelines using Delta Lake and Medallion Architecture principles.
- Implement Delta Lake features including ACID transactions, schema evolution, Change Data Feed (CDF), time travel, and Liquid Clustering.
- Build real-time streaming pipelines using Structured Streaming, checkpoints, and event-time processing.
- Implement enterprise data governance using Unity Catalog, storage credentials, external locations, data lineage, row filters, and column masking.
- Secure Azure Databricks environments using Microsoft Entra ID, Azure Key Vault, cluster policies, and role-based access control.
- Create and automate production data engineering workflows using Databricks Jobs and Lakeflow Jobs.
- Optimize Apache Spark workloads using Photon, Adaptive Query Execution (AQE), partitioning, caching, and query optimization techniques.
- Monitor and troubleshoot Azure Databricks workloads using Spark UI, Azure Monitor, query history, and cluster diagnostics.
- Integrate Azure Databricks with Azure Data Factory, Azure Event Hubs, Azure Monitor, Azure Key Vault, GitHub, and Azure DevOps.
- Design secure, scalable, and governed enterprise Lakehouse solutions.
- Prepare for Exam DP-750 and earn the Microsoft Certified: Azure Databricks Data Engineer Associate certification.
Pre-requisites
Participants should have:
- Basic understanding of data engineering concepts and ETL processes.
- Working knowledge of relational databases and SQL.
- Familiarity with Microsoft Azure fundamentals and cloud computing concepts.
- Basic programming knowledge in Python and SQL.
- Understanding of Azure Storage services such as Azure Data Lake Storage Gen2.
- Familiarity with data transformation concepts and distributed data processing is recommended.
- Prior exposure to Apache Spark or big data technologies is beneficial but not mandatory.
- Experience with Azure Data Factory or modern analytics platforms is advantageous.
- Knowledge equivalent to Microsoft Azure Fundamentals (AZ-900) is recommended.
Course Curriculum
-
-
-
- Overview of Databricks Lakehouse Platform
- Understanding modern data engineering
- Azure Databricks architecture
- Apache Spark overview
-
-
- Workspace navigation
- Cluster creation and configuration
- Interactive and job clusters
- Notebook fundamentals
- Using Python, SQL, and Spark commands
- Collaborative notebook development
- Visualizations and widgets
- Version control basics
- Create Databricks workspace
- Configure clusters
- Execute Spark notebooks
- Run SQL and DataFrame operations
- Batch vs streaming ingestion
- File formats:
- CSV
- JSON
- Parquet
- Delta
- Data ingestion from Azure Storage
- Auto Loader architecture
- Schema inference and evolution
- Incremental file processing
- Spark DataFrames
- Spark SQL transformations
- Joins, aggregations, and filtering
- Data cleansing techniques
- Build ingestion pipelines
- Use Auto Loader
- Perform transformations using Spark SQL
- Process raw data into curated datasets
Module 7: Delta Lake Fundamentals
- Delta Lake architecture
- ACID transactions
- Schema enforcement and evolution
- Time travel and rollback
- Data compaction
- Z-ordering
- Vacuum operations
- Table optimization strategies
- Streaming architecture
- Stream processing concepts
- Watermarking and checkpoints
- Real-time analytics pipelines
- Create Delta tables
- Implement streaming pipelines
- Perform time travel queries
- Optimize Delta Lake performance
- Databricks Jobs
- Task orchestration
- Scheduling workflows
- Monitoring and retry strategies
- Unity Catalog architecture
- Data governance principles
- Catalogs, schemas, and tables
- Access control management
- Role-based access control
- Secret scopes
- Secure cluster configuration
- Compliance and governance
- Create and schedule jobs
- Configure Unity Catalog
- Implement permissions and access policies
- Secure notebooks and data assets
- Cluster sizing strategies
- Query optimization
- Partitioning and caching
- Shuffle optimization techniques
- Job monitoring
- Cluster diagnostics
- Debugging Spark applications
- Failure recovery
- Azure Data Factory integration
- Azure ML integration
- End-to-end data engineering architecture
- Certification exam overview
- Important exam domains
- Sample case studies and scenarios
- Practice discussions and Q&A
- Build an end-to-end data engineering pipeline using Azure Databricks
