Skip to main content

Databricks

How to connect to Databricks​

This guide explains how to connect to tables and views directly within Databricks or connect to Databricks tables and views from Power BI. Databricks operates as a modern data warehouse, which serves as the single source of truth for the data within ESG.

Prerequisites​

Before connecting, ensure that:

Organization of catalogs​

There are 3 main catalogs within Databricks:

  • Bronze: an abstraction layer over the files we extract and store in an Azure Data Lake Gen2 Storage Account
  • Silver: data already in a cleaned format
  • Gold: data modelled in dimensions and facts, that will be used for connecting to Power BI

We advise to always work on the gold catalog in the production environment (PRD_GOLD) since these tables and views will have transformed data that has been validated.

SQL Editor in Databricks​

Once in the SQL Editor, which can be found in the left navigation pane, you can choose to create a new query and write content within this query editor to access data of the bronze/silver/gold tables. You can also save or share a query so that it can be (re)used by other people within the organization.

Power BI on Databricks​

The majority of tables are modeled until the gold layer of Databricks, from which a Power BI dataset will be created. In case you want to access these tables within prepared datasets, please consult the Power BI page.

However, within Power BI, you can also connect directly to Databricks on the layer that you want (bronze/silver/gold).

  • Open Power BI desktop.
  • Create a blank report.
  • Connect to the Databricks Production environment: Home → Get Data → Azure Databricks
  • Enter the Server Hostname (adb-2603486107066893.13.azuredatabricks.net) and HTTP Path (sql/protocolv1/o/2603486107066893/0616-143805-isi44hgm), leave all the other fields untouched, and click on 'OK'.
  • Select the tables or views that you need from the appropriate catalog, and click on 'Load'.