Spotify End-To-End Azure Data Engineering Project (From Beginner To Pro)

Azure End To End Data Engineering Project | Azure Data Factory | Azure Databricks | Azure SQL DB | PySpark | Big Data 🔍What You'll Learn? In this 7-hour in-depth tutorial, you’ll build a complete Data Engineering project from scratch using powerful tools like Azure Data Factory, Azure SQL DB, Azure Databricks, Unity Catalog, Delta Live Tables, Spark Streaming, PySpark, Databricks Asset Bundles, GitHub, and more. You’ll also dive into real-world concepts like Dimensional Data Modeling and Slowly Changing Dimensions (SCDs). Timestamps: 0:00 Introduction 05:53 Azure Data Project Architecture 17:16 Free Azure Account Creation 20:25 Azure Portal Overview 28:23 Creating Azure Resources 47:51 Azure Data Factory Tutorial 1:04:12 Azure SQL DB Source 1:17:12 Incremental Ingestion Pipelines With Data Factory 2:06:00 Adding Backfilling Feature In The Pipelines 2:20:41 Looping The Pipelines 2:44:25 Logic Apps With Azure Data Factory 3:16:29 Azure Databricks Tutorial 3:44:56 Databricks Unity Catalog Tutorial 3:59:06 Spark Streaming With Databricks Autoloader 4:20:58 Python Utilities 4:30:38 PySpark Transformations & APIs 5:07:40 Metadata Driven Pipelines With PySpark and Jinja2 5:37:48 Star Schema and Slowly Changing Dimensions 5:50:47 Databricks Delta Live Tables Tutorial 6:27:12 Databricks Incremental Data Load 6:38:26 Databricks Asset Bundles For CI/CD WATCH MY PREVIOUS VIDEOS ➡️PySpark Full Course https://youtu.be/94w6hPk7nkM?si=i3u_8ZWkGf6fiGyh ➡️ DBT Masterclass https://youtu.be/B8uwFmVt4sU?si=lkEwzsA5TucQdx7u ➡️Azure End To End Data Project https://youtu.be/ESWqAZP2qA4?si=eWBmBQYOk5pMYfxh ➡️Databricks Tutorial https://youtu.be/7pee6_Sq3VY?si=pPBrTnE2R8PhhHu5 ➡️Apache Spark The Ultimate Guide https://youtu.be/FNJze2Ea780?si=MHszd1EpzUtFzZR9 REPOSITORY ➡️GitHub https://github.com/anshlambagit/spotify_azure_project ======================================= Join this channel to SUPPORT MY HARDWORK: https://www.youtube.com/channel/UCu7lQE-L5gzt8aD7zuuufjw/join ======================================= Connect with ME LinkedIn - https://www.linkedin.com/in/ansh-lamba-793681184/ Telegram - https://t.me/anshlambadatafam ========================= For COLLABORATION 👇 anshlambacollab@gmail.com ========================= ⭐Hashtags⭐ #azure #azuredatafactory #databricks #pyspark #dataengineering