Databricks Setup

Rohit K
Rohit K
  • Updated

Overview

The Databricks Synapse enables data synchronization between your Databricks tables and Syncari entities using databricks SDK.  After the Synapse is authenticated, a user can activate the Databricks Synapse, which enables Syncari to replicate the schema of Databricks and make it available within the Unified Data Model. Using the Sync Studio, a user can map Databricks tables to Syncari entities, and table fields to Syncari fields.

Note:

Supports

  1. Watermark Entities
  2. Non watermark entities
  3. Composite Id

After setting up 

  1. Entities user is planning to map into Syncari. Please select id field for that entity in Databricks Synapse and publish it.

  2. Entities that the user plans to map into Syncari

    1. By default, entities include the field syncari_watermark_timestamp. Do not delete this field.

    2. If you choose to use a different watermark field, you must update the entity to deselect syncari_watermark_timestamp and select the new field instead.

    3. Please select the correct watermark field for each entity in Databricks Synapse and publish it

    4. If no field is selected, the entity will behave as if it has no watermark, meaning all records will sync every time.

Prerequisites

  • Access to a Databricks workspace
  • A running or available SQL warehouse in that workspace
  • Workspace admin permissions (or an admin available to assist)

Part 1 - Setup in Databricks

Step 1: Gather your connection details

1. In your Databricks workspace, copy the base URL from the browser address bar. This is your Workspace URL (for example, https://dbc-5f20fe00-bf94.cloud.databricks.com). 
Note: Remember to add with https://

2. Go to SQL Warehouses in the left sidebar and select the warehouse you want Syncari to query through. 

Open the Connection details tab. Copy the value from the HTTP path field, the segment after /warehouses/ is your SQL Warehouse ID.

3. Go to Catalog in the left sidebar and identify the catalog and schema you want Syncari to sync with. Note the exact names, as they must match what you enter in Syncari.

Step 2: Create a service principal and generate credentials

1. In Databricks, go to your profile, open Settings.

2. Go to  Identity and access. For Service principals, click Manage.

3. Click Add service principal to workspace, then in the pop-up box, click the drop-down and select Add new service principal to account and workspace.

4. Name the service principal to reflect its purpose (for example, syncari-integration), and then click Add Service Principal.

5. Open the service principal's details page and go to the Secrets tab. Click Generate secret.

7. Add the number of days for the new secret to be valid and select a scope when generating the secret, choose sql. Then click Generate. 

8. Copy the Secret and Client ID values shown. Store them securely.

Step 3: Grant Permissions

1. Go to your SQL warehouse and click Permissions, and in the pop-up box, add your service principal and grant the service principal Can Use.

2. In Catalog section, select the target catalog (for example, workspace) and open its Permissions tab. Click Grant.

Now add the service principal as the principal, and select USE CATALOG from the privileges list. Then, click Confirm.

3. Repeat this on the target schema (for example, workspace > default): open its Permissions tab, click Grant, and select USE SCHEMA (or broader SELECT) for the service principal. Then click Confirm.

Part 2: Set Up the Synapse in Syncari

1. Log in to your Syncari subscription and navigate to Synapses.

2. Find Databricks in the Synapse library and drag it onto the canvas.

3. Configure your Synapse parameters:

  • Enter [Synapse Name]
  • Enter [Workspace URL] - Please  make sure to add prefix https:// for url
  • Enter [SQL Warehouse ID] 
  • Enter [Catalog]
  • Enter [Schema]
  • Click Authenticate.

4. Add the Client ID and Client Secret we generated in Databricks. Then click Authenticate.

5. Once Authenticated, click Finish.

6. Then click Make Synapse Active.

 

Congratulations! You have completed the setup of Databricks.

If you receive any errors message after clicking Authenticate, confirm each of the parameters in the previous steps are configured properly. If the error persists, contact one of our Onboarding and Support team members.

Share this

Was this article helpful?

0 out of 0 found this helpful