Azure Synapse Analytics - Easy Fix AVAILABLE_COMPUTE_CAPACITY_EXCEEDED by creating new Spark Pool

 In this video, we will see how you can fix the error AVAILABLE_COMPUTE_CAPACITY_EXCEEDED that you will generally face when you are trying to create a new spark pool and using same pool to another notebook. In this video, to fix that issue we will create a new pool by requesting for vCores

Quota Documentation :

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-concepts

To submit ticket:

https://learn.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/sql-data-warehouse-get-started-create-support-ticket

ENROLL NOW TO LEARN AZURE SYNAPSE ANALYTICS FROM SCRATCH!! 

Couse link [WITH COUPON CODE] : https://www.udemy.com/course/basics-to-advanced-azure-synapse-analytics-hands-on-project/learn/lecture/39340886?couponCode=FIRST70

What you'll benefit from this course?

1. Synapse concepts:  Complete basic to advanced understanding of Azure Synapse Analytics.

2. PySpark from Scratch: Explore 50+ essential PySpark Transformations that power data manipulation and enrichment.

3. Access to 45 + Spark notebooks : Acquire a comprehensive library of 45+ PySpark notebooks for efficient data cleansing and transformation.

4. Build entire project with Synapse: Dive into hands-on learning, building a modern data warehouse using Azure Synapse.

5. Python for Data Engineers: Discover the role of Python in Data Engineering and its crucial applications.

6. Spark pool for data analysis: Harness the capabilities of Spark Pools for processing large-scale data workloads.

7. Spark Optimization techniques: Optimize your PySpark skills with Spark Optimization techniques for enhanced performance.

8. Delta lake and its features: Uncover the principles and advantages of Delta Lake as a reliable data storage and management solution.

Read More

Azure Synapse Analytics - Course Architecture - Build complete project only using synapse and PySpark!

 

Build entire project using Azure Synapse Analytics focused on PySpark

Course link [With coupon Code] : https://www.udemy.com/course/basics-to-advanced-azure-synapse-analytics-hands-on-project/learn/lecture/39340886?couponCode=FIRST70

What you'll learn from this course?

1. Synapse concepts: Complete basic to advanced understanding of Azure Synapse Analytics.

2. PySpark from Scratch: Explore 50+ essential PySpark Transformations that power data manipulation and enrichment.

3. Access to 45 + Spark notebooks : Acquire a comprehensive library of 45+ PySpark notebooks for efficient data cleansing and transformation.

4. Build entire project with Synapse: Dive into hands-on learning, building a modern data warehouse using Azure Synapse.

5. Python for Data Engineers: Discover the role of Python in Data Engineering and its crucial applications.

6. Spark pool for data analysis: Harness the capabilities of Spark Pools for processing large-scale data workloads.

7. Spark Optimization techniques: Optimize your PySpark skills with Spark Optimization techniques for enhanced performance.

8. Delta lake and its features: Uncover the principles and advantages of Delta Lake as a reliable data storage and management solution.

Read More

Introducing my new course: "Basics to Advanced: Azure Synapse Analytics Hands-On Project" 📊

 

🚀 Exciting News! Introducing my new course: "Basics to Advanced: Azure Synapse Analytics Hands-On Project" 📊

🔥 Unlock the Power of Azure Synapse Analytics! 🚀

📚 18.5+ HOURS OF IN-DEPTH LEARNING CONTENT! 📚

🔍 Ready to dive into the world of data analytics and unleash the true potential of Azure Synapse Analytics? Join me on a transformative learning journey where you'll gain a complete understanding of Azure Synapse Analytics, from its fundamental principles to advanced techniques.

🔥 Here's what you'll experience in this comprehensive course:

1. 🧠 Complete basic to advanced understanding of Azure Synapse Analytics.
2. ✨ Explore 50+ essential PySpark Transformations that power data manipulation and enrichment.
3. 📓 Acquire a comprehensive library of 45+ PySpark notebooks for efficient data cleansing and transformation.
4. 🏗️ Dive into hands-on learning, building a modern data warehouse using Azure Synapse.
5. 🐍 Discover the role of Python in Data Engineering and its crucial applications.
6. 💡 Harness the capabilities of Spark Pools for processing large-scale data workloads.
7. ⚙️ Optimize your PySpark skills with Spark Optimization techniques for enhanced performance.
8. 🌊 Uncover the principles and advantages of Delta Lake as a reliable data storage and management solution.

Are you ready to transform your data analytics journey? Don't miss out on this opportunity to elevate your skills and propel your career forward. Enroll today and unlock the endless possibilities that Azure Synapse Analytics has to offer!

Click on the link coupon link:

👉 https://www.udemy.com/course/basics-to-advanced-azure-synapse-analytics-hands-on-project/?couponCode=FIRST70

Read More

Azure Synapse Analytics - Can 2 different spark notebooks connect to a same spark pool and execute in parallel?

 

Azure synapse analytics – Spark

 

FAQ  1- Can 2 different spark notebooks connect to a same spark pool and execute in parallel?

 

Answer: Yes

I’m having workspace capacity of 80vCores

Taking an example, if you have created a spark pool of Node size: Small (4 vCore – 32 GB size) with 8 nodes.

Total pool size = 32vCores

You can set the number of nodes to be used for each notebook

Notebook 1: Total 3 nodes = (1 driver node and 2 executor nodes)

4vCores x 3 nodes = 12 vCores used

 

Notebook 2:

Total 4 nodes (1 driver node and 3 executor nodes)

4vCores x 4 = 16 vCores used

 

 

12 vCores + 16 vCores = 28 vCores

 

Total of pool size with 32vCores, 28vCores were utilized which is 87.5% utilization.

You can run 2 notebooks having a single Spark pool

 

FAQ 2: Can these share the variables or Temporary views created in their notebooks as they are attached to same pool?

Answer: No

 

Explanation:

Apache Spark for Synapse is designed as a job service and not a cluster model. It creates a separate Apache Spark application to run each notebook.

 

 

 

Read More

Azure Data Factory + Azure Synapse Analytics - END to END development Project course - Grab 50% OFF COUPON and ENROLL NOW!

Coupon Code link: 

https://www.udemy.com/course/azure-data-factory-synapse-analytics-end-to-end-etl-project/?couponCode=NEWJUNE60

With 450+ Students join this course with above Link for can access the course with 50% OFF!!!

Throughout this course, you'll gain practical hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You'll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like 

1. Azure SQL Database

2. Azure Synapse Analytics 

3. Azure Key Vault 

4. Azure Data Factory for Orchestration,

5. Azure Storage solutions (Azure Datalake Gen2)

6. Microsoft Power BI

7. Azure Logic Apps

============================

Linkedln : https://www.linkedin.com/in/shanmukh-sattiraju/

View the below video for project architecture:

Read More

Azure Synapse Analytics - Reading files from Azure Datalake and Writing to ADLS using PySpark

 Accessing storage account from Azure Synapse Analytics

This can be directly accessed using Linked service, 

With linked service we can access by "Account key" or by "User assigned Managed Identity"

Microsoft reference Documentation link: 

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-python

Read More

Power BI - Use columns as filters and dynamically change data of the visuals without bookmarks !

 In this video we are making use of columns as filters without using any bookmarks or buttons to change the data inside the visuals. 

We are making use of Field Parameter to change the visuals dynamically. This will help us to reduce the effort of using bookmarks if we have multiple values to filter or changes in a specific visual.

Read More

New course on Complete end to end ETL project on OTT Platform using Azure Data Factory + Azure Synapse Analytics from Scratch!!

 Glad to announce my First Udemy Course on Building an end to end  ETL project on OTT platform using Azure Data Factory and Azure Synapse Analytics !!

150+ ENROLMENTS IN 48 HOURS!!!

Udemy course with Coupon link (50% DISCOUNT PRICE !!!):

https://www.udemy.com/course/azure-data-factory-synapse-analytics-end-to-end-etl-project/?couponCode=A8E8D451E891AF2C3C46

Limited time to enrol this course with above Link for can access the course with 50% OFF!!!

Throughout this course, you'll gain practical hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You'll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like 

1. Azure SQL Database

2. Azure Synapse Analytics 

3. Azure Key Vault 

4. Azure Data Factory for Orchestration,

5. Azure Storage solutions (Azure Datalake Gen2)

6. Microsoft Power BI

7. Azure Logic Apps

Read More

Azure Data Factory- Execute Synapse Analytics notebook from ADF pipeline- New feature!!

 We recently have seen ADF added a new feature to call the notebook of Synapse analytics from Azure Data Factory.

We have a notebook activity been added in Azure Data Factory, in this video we are going to see how to run that notebook from ADF

Read More

Power BI - How to generate Fiscal Year columns in Date Table using DAX Formulas

Here we are going to see How to generate Fiscal Year columns in Date Table in Power BI using DAX.

We are seeing how to create Fiscal Year Number, Fiscal Month, Fiscal Month Number, Fiscal Quarter, Fiscal Year with Quarter, Fiscal Year With Month.

 

Read More

Azure DevOps - Quick Solution - "No hosted parallelism has been purchased or granted Error" --Fixed!

You will not be given any free grant of agents by default from Microsoft with its new policy

To build your code or deploy your software using Azure Pipelines, you need at least one agent. As you add more code and people, you'll eventually need more.

When your pipeline runs, the system begins one or more jobs. An agent is computing infrastructure with installed agent software that runs one job at a time.

2 types of Agents:

1.      Microsoft Hosted Agents

2.      Self-Hosted Agents

Microsoft Hosted: Here the will host the agent

Self-hosted – You can host the agent to run the job for you

 

If you do not want to wait for 2 business days and if you are doing these tests you can host a self hosted agent in a virtual machine or in your own local machine

Read More

Azure Data Factory - Setup Continuous Integration Continuous Delivery (CI/CD) using Azure DevOps Git

 In this video, we are configuring the Continuous Integration and Continuous Delivery (CICD) from scratch in ADF using Azure DevOps Git.

Read More

Azure Data Factory - Copy multiple SQL tables incrementally using a watermark table (Delta Load) ADF

 Azure Data Factory - Copy multiple SQL tables incrementally using a watermark table (Delta Load) ADF

Here we are doing incremental copy of multiple SQL Tables from one Database to another Database. 

Please refer the below video Step by Step explanation of pipeline for a Single SQL Database:

https://youtu.be/AOClU3s9jXw

We connected to SQL Database from On-Premise Environment for this video. To know how to setup Self Hosted Integrated Runtime,

Please refer:

https://youtu.be/d9Xp2pnjYcI

Read More

Azure Data Factory - Copying files from On premise Linux or Unix Server to Azure Datalake using ADF

 Here we are copying files from Linux Virtual Machine to ADLS. We are created a file share from linux and we are adding that file share to windows, 

using self hosted IR we are accessing the Linux File Share and we are able to copy the files from Linux to Azure DataLake

Link for configuring Self Hosted Integration Runtime to access On premise Windows file:

https://youtu.be/d9Xp2pnjYcI

To install samba utility:

-- sudo apt install samba -y

To start samba service:

-- sudo systemctl enable --now smbd

To check status of service:

-- sudo systemctl status smbd

To create a samba user:

-- sudo smbpasswd -a (username)

Configuring Samba share from Linux to Windows

Copy files from Linux to Azure Datalake using ADF

Read More

Azure Data Factory - Status of Trigger runs during their pipeline failure situation

 Azure Data Factory - Status of Trigger runs during their pipeline failure situation

Here we are seeing behavior of Tumbling Window, Scheduled and Storage Event triggers during the event of pipeline failure.

 

To configure Self Hosted Integration Runtime:

https://youtu.be/d9Xp2pnjYcI

Read More

Azure Data Factory -Delete files based on Last modified date (delete more than 7 days ) using ADF

Azure Data Factory -Delete files from ADLS Gen2 based on Last modified date (delete more than 7 days ) using ADF

In this we are deleting files from Azure Data Factory based on the last modified date (as per requirement) , using this we can perform clean up for unwanted or stale data. 

Read More

Azure Data Factory - Create a Customer Managed Key and encrypt ADF using that CMK

Azure Data Factory - Create a Customer Managed Key and encrypt ADF using that CMK

By default, data is encrypted with a Microsoft-managed key that is uniquely assigned to your data factory. For extra security , you can use customer-managed keys feature in Azure Data Factory. 

When you specify a customer-managed key, Data Factory uses both the factory system key and the CMK to encrypt customer data. Missing either would result in Deny of Access to data and factory.

Reference Documentation Link:  https://docs.microsoft.com/en-us/azure/data-factory/enable-customer-managed-key

Read More

Azure Data Factory - Send an alert email when a blob is created or deleted from ADLS Gen2 using ADF

 Azure Data Factory - Send an alert email when a blob is created or deleted from ADLS Gen2 using ADF

Notifying about the alerts is an important part in ADF, we can achieve that by following the steps in the video. Here we are getting email alerts to GMAIL whenever a file is removed and created in our Storage account.

Refer the below video to get custom alert email when pipeline failed : https://youtu.be/KorFyv5FntY

Read More

Azure Data Factory - Import or Copy Pipeline from One Data Factory to Another Data Factory

 Azure Data Factory - Import or Copy Pipeline from One Data Factory to Another Data Factory

Here we are importing one ADF to another ADF, all the configurations in one pipeline will be copied to another using this

Read More

Azure Data Factory - Copy multiple files from HTTP website dynamically to Data Lake using ADF

 Azure Data Factory - Copy multiple files from HTTP website dynamically to Data Lake using ADF

Here we are copying multiple files dynamically from HTTP website . We are using GitHub as source website to copy multiple files to Azure Data Lake Gen2

Previous video to copy single HTTP file: https://youtu.be/PNN5VPoP2zQ

Read More

Azure Data Factory - Copy data from HTTP website (GitHub) to Azure Data lake using ADF

 Azure Data Factory - Copy data from HTTP website to Azure Data lake using ADF

Here we are copying a csv file from a HTTP website (GitHub) to Azure Data lake Storage using copy activity from ADF

Refer the below link for copying multiple files from a HTTP website dynamically to ADLS:  https://youtu.be/K5ND-pyD3yE

Read More

Azure Data Factory - Incremental load or Delta load using a watermark Table

Here source is having incoming values with some date value, we are keeping a watermark table which keeps track of the data copied and this helps to incrementally load the data to the destination preventing copy of the old data.

Refer below to setup self hosted IR to use SQL from local Machine:

https://youtu.be/d9Xp2pnjYcI

Read More

Azure Data Factory - Copy multiple files from AWS S3 to Azure Data Lake Storage using ADF pipeline

 Azure Data Factory - Copy multiple files from AWS S3 to Azure Data Lake Storage using ADF pipeline

Here we are getting all the files in a folder in Amazon S3. We are copying all files dynamically using Azure Data Factory

To prevent hard coded values in sensitive information, its recommended to use Azure Key  Vault. Refer the below video for reference to how to use Azure Key vault in ADF: 

https://youtu.be/2lPlReATWew

Read More

Azure Data Factory - Delete specific folders and specific files using lookup file in ADF

 Azure Data Factory - Delete specific folders and specific files using lookup file in ADF

Here we are making a lookup file as reference which tells the specific files and folders to be deleted. We are using lookup , forEach and delete activity to execute this logic to delete the required files

Read More

Azure Data Factory - Create Managed VNET Integration runtime and creating Managed Private Endpoint

Azure Data Factory - Create Managed VNET Integration runtime and creating Managed Private Endpoint

In this we are first creating Managed VNET Integration runtime and we are establishing a private endpoint connection with ADLS Gen2 and doing a copy activity using Managed VNET Integration Runtime

Read More

Azure Data Factory - Copy specific range of cells in Excel along with sheets dynamically to CSV

 Azure Data Factory - Copy specific range of cells in Excel along with sheets dynamically

Here we are copying only specific range of cells from an Excel file , using this we can eliminate unwanted cells from an Excel while copying

Below is the video which tells about copying excel sheets dynamically in details

https://youtu.be/n5phQgvIaxY

Read More

Azure Data Factory - Moving Files from Source to Destination in ADLS using copy and delete Activity

In Azure Data Factory - We do not have any move activity , we can take advantage of copy and delete activity to fulfill the requirement of moving.

In this video we are moving all the csv files from one folder to another

Read More

Azure Data Factory - Copy multiple sheets in excel dynamically into multiple files/tables /SQL table

 Azure Data Factory - Copy multiple sheets in excel dynamically into multiple files/tables /SQL table

We are copying multiple sheets in Excel dynamically into multiple Files, which created each csv file for a sheet.

We are also copying these sheets into a SQL table and also multiple table based on the name of the sheet

Read More

Azure Data Factory - Add additional column dynamically with Row count of files in Copy activity

 Add additional column dynamically with Row count of files in Copy activity

In this we are dynamically adding a column which tells us the row count that each file holds in the Azure Data Lake and we are copying them using Copy activity to SQL Database

Read More

Azure Data Factory - Adding Keyvault secret for password in SSIS Package in pipeline

 Azure Data Factory - Adding Keyvault secret for password in SSIS Package in Azure Data Factory

Storing the sensitive information is an importing thing while developing a solution. Using Azure KeyVault we can store our secrets and access them using a linked service in Azure Data Factory.

Read More

Azure Data Factory - Run SSIS packages in ADF as File System (Project Model) using SSIS-IR

 Azure Data Factory - Run SSIS packages in ADF as File System (Project Model) using SSIS-IR

In this we are using Project Deployment Model, we are taking the .ispac file that is getting generated after build. From that we are uploading the .ispac file and the package(.dtsx) to Azure File Share. 

If you want to know the FileSystem (Package),

Please visit: https://youtu.be/LzvoUwriio8

Read More

Azure Data Factory - Run SSIS packages in ADF as File System (Package Model) using SSIS-IR

 Azure Data Factory - Run SSIS packages in ADF as File System (Package Model) using SSIS-IR

In this we are using package deployment model and storing our Configuration File (.dtsConfig) and Package(.dtsx) in an Azure File Share.  

Using Azure Data Factory we are calling the package as File System (Package) and giving the configuration file with package

ADF looks for the configuration file and performs the action based on the content /variables that we given in the configuration file

Read More

Azure Data Factory -Run SSIS-IR using Self Hosted IR as proxy to access On premise SQL Database Demo

 Azure Data Factory -Run SSIS-IR using Self Hosted IR as proxy to access On premise SQL Database Demo

Here we are accessing Onpremise SQL Database Data using Self Hosted IR as a proxy, We are deploying the SSIS-IR in the Azure SQL Database.

Read More

Azure Data Factory - Lift & Shift SSIS -Execute SSIS package in ADF using Project deployment Model (SSISDB)

Azure Data Factory - Lift & Shift SSIS -Execute SSIS package in ADF using Project deployment Model (SSISDB) by creating SSIS - Integration Runtime

In this video we are using Project Deployment model . We are deploying our Project into SSISDB which is being Hosted in Azure SQL Database Server. 

We are executing the packages from Azure Data Factory which are hosted in the SSISDB.

Read More

SSIS - Deploy SSIS-DB Catalog in local SQL Server - Project Deployment Model

 Here we are deploying the SSISDB catalog into on premise local SQL server

Read More