Dataset as Source

Varsha Neelesh
Varsha Neelesh
  • Updated

Datasets created in Insights can now be used as sources in sync studio pipelines to sync data. These are especially useful when you want to sync calculated data back into your operational systems. It is also a great way to create daily/weekly/monthly snapshot of data.

 

Usage

 Datasets that are created in Insights are automatically visible in source entities within sync studio.

Screenshot 2024-01-23 at 1.19.38 PM.png

They can be used as source entities, like all other synapse entities. The id field needs to be explicitly defined for a dataset via schema studio. The watermark field is automatically assigned by Syncari (Updated At). 

When the pipeline runs, all records from the dataset are read in every sync cycle.

Screenshot 2024-01-23 at 1.20.01 PM.png

Dataset used in a pipeline cannot be deleted via insights unless, all dependencies from pipelines are removed. Any edits to the dataset is automatically applied to the dataset used in pipelines.

Share this

Was this article helpful?

0 out of 0 found this helpful