Azure Data Engineering Cookbook - Second Edition

Over 90 recipes to help you collect and transform data from multiple sources into a single data source which makes it easier to perform analytics on the data Key Features * Build data pipelines from scratch and find solutions to common problems in data engineering lifecycle * Learn how to work with Data Factory Azure Data Lake Azure Databricks and Azure Synapse Analytics * Monitor and troubleshoot your data engineering pipelines using log analytics and Azure monitor Book Description Data is the new oil and gaining maximum insights out of data is extremely critical for an organization's success. Building performant data engineering pipelines to ingest store process and visualize data is one of the major challenges organizations face in leveraging value out of data. This book shares 90 useful recipes covering common scenarios in building data engineering pipelines in Azure. The book a second edition of the immensely successful first edition written by Ahmad Osama covers several recent enhancements in Azure data engineering. This edition explores recipes from Azure Synapse Analytics workspaces - gen 2 covering topics like Synapse spark pools SQL Serverless pools Synapse integration pipelines and synapse data flows. The book also dives deep into Synapse SQL Pool optimization techniques in the second edition. Besides Synapse enhancements it covers building the semantic and visualization layer using Power BI and establishing connectivity of Databricks and Synapse pools with Power BI. Finally the book covers overall data engineering pipeline management focusing on areas like tracking impact and data lineage using Azure Purview. By the end of this book it will serve as your go-to guide in building excellent data engineering pipelines. What you will learn * Perform data ingestion and orchestration using Azure Data Factory * Move data from on-premise sources to Azure using Data Factory Integration Runtime * Process your raw data using Azure Databricks and Azure Synapse * Perform data orchestration and ETL tasks using Azure Synapse analytics * Implement high availability and monitor performance of Azure SQL Database * Build effective Synapse SQL pools which can be consumed by Power BI * Monitor the performance of Synapse SQL and Spark pools using log analytics Who This Book Is For This book is targeted at Data engineers Data architects database administrators and data professionals who want to get well versed with the Azure data services for building data pipelines. Basic understanding of cloud and data engineering concepts would be beneficial.