BigQuery Setup

The BigQuery Synapse enables data synchronization between the BigQuery schema and Syncari entities using the Google API. After the Synapse is authenticated, a user can activate the BigQuery Synapse, which enables Syncari to replicate the schema of BigQuery and make it available within the Unified Data Model. Using the Sync Studio, a user can map BigQuery tables to Syncari entities, and table fields to Syncari fields.

If your BigQuery security requires you to whitelist an IP you can find that here.

Required BigQuery / Google Cloud Platform Information

You will need the following information from your Google Cloud Platform Project and BigQuery dataset. 

• The Project ID from your Google Cloud Platform project
• A Service Account with a JSON Key assigned to the Service Account
  • Use an existing Service Account or create a new one related to your Project under IAM > Service Accounts.
• The BigQuery Dataset ID
  
  • Each Dataset ID will need to be configured with a new Synapse connection.

Important Information

• bigquery.jobs.create permission should be assigned to the service account. This is required to enable Syncari to read from BigQuery datasets. 
• Using BigQuery tables as destinations
  • BigQuery keeps values from INSERT/UPDATE operations in a stream and it may take about 90 minutes for the stream to be committed to the tables. While you can still query data in the stream, you cannot issue updates to these records, until they are committed to the main table. So, consecutive updates to the same record in Syncari, within that 90 minute window may result in Syncari missing the second (and later) updates to the record.
  • We recommend setting the pipelines to sync at most once per 90 minutes to minimize the impact of this behavior. We are exploring other ways to mitigate this issue, and will update this documentation accordingly.
• Spaces in table names and column names are currently unsupported. We recommend renaming your tables/columns and exclude the use of spaces in their names. 

Synapse Setup

Step 1: Start by logging in to your Syncari subscription

1. Navigate to Synapses
2. Find BigQuery in the Synapse library
3. Drag the BigQuery Synapse onto the canvas

Step 2: Configure your Synapse parameters

1. Enter [Synapse Name]
2. Enter [Project ID]
3. Enter [Dataset ID]
4. Click "Next"

Step 3: Set your JSON Key and authenticate:

1. Enter [Key]
   This is your JSON Key for the Service Account that's related to the Project.
2. Click "Save and Authenticate"
3. Click "Next"

Step 4: Activate your Synapse

1. Click "Make Synapse Active"
2. Click "Close"

Congratulations! You have completed the setup of BigQuery.

If you receive any errors message after clicking Authenticate, confirm each of the parameters in the previous steps are configured properly. If the error persists, contact one of our Onboarding and Support team members.

NOTE: When waiting for incremental sync some records may take up to 90 minutes to arrive. This is due to the nature of the BigQuery streaming API.

What's Next?

• Learn how to designate the Id and Watermark fields for BigQuery Entities.