Scale Azure Databricks secure network access to Azure Data Lake Storage

Many organisations store their data in a centralised data lake on Azure Storage accounts. In this post we will look at a solution to scale the secure network access between the centralised data lake storage account and Azure Databricks workspaces VNets. The recommended approach to achieve a scalable and secure network access to the data lake storage is to use Private Endpoints for the storage accounts. Private Endpoint relies upon DNS resolution to automatically route the connections from the VNet to the storage account over a private link. Read more