The BigQuery Synapse enables data synchronization between the BigQuery schema and Syncari entities using the Google API. After the Synapse is authenticated, a user can activate the BigQuery Synapse, which enables Syncari to replicate the schema of BigQuery and make it available within the Unified Data Model. Using the Sync Studio, a user can map BigQuery tables to Syncari entities, and table fields to Syncari fields.
If your BigQuery security requires you to whitelist an IP you can find that here.
BigQuery / Google Cloud Platform Information
You will need the following information from your Google Cloud Platform Project and BigQuery dataset.
- The Project ID from your Google Cloud Platform project
- A Service Account with a JSON Key assigned to the Service Account
- Use an existing Service Account or create a new one related to your Project under IAM > Service Accounts.
- The BigQuery Dataset ID
- Each Dataset ID will need to be configured with a new Synapse connection.
Step 1: Start by logging in to your Syncari subscription
- Navigate to Synapses
- Find BigQuery in the Synapse library
- Drag the BigQuery Synapse onto the canvas
Step 2: Configure your Synapse parameters
- Click "Next"
Step 3: Set your JSON Key and authenticate:
This is your JSON Key for the Service Account that's related to the Project.
- Click "Save and Authenticate"
- Click "Next"
Step 4: Activate your Synapse
- Click "Make Synapse Active"
- Click "Close"
Congratulations! You have completed the setup of BigQuery.
If you receive any errors message after clicking Authenticate, confirm each of the parameters in the previous steps are configured properly. If the error persists, contact one of our Onboarding and Support team members.
NOTE: When waiting for incremental sync some records may take up to 90 minutes to arrive. This is due to the nature of the BigQuery streaming API.
- Learn how to designate the Id and Watermark fields for BigQuery Entities.
Caveat when using BigQuery tables as destinations
BigQuery keeps values from INSERT/UPDATE operations in a stream and it may take about 90 minutes for the stream to be committed to the tables. While you can still query data in the stream, you cannot issue updates to these records, until they are committed to the main table.
So, consecutive updates to the same record in Syncari, within that 90 minute window may result in Syncari missing the second (and later) updates to the record.
We recommend setting the pipelines to sync at most once per 90 minutes to minimize the impact of this behavior. We are exploring other ways to mitigate this issue, and will update this documentation accordingly.