Datasets created in Insights can now be used as sources in sync studio pipelines to sync data. These are especially useful when you want to sync calculated data back into your operational systems. It is also a great way to create daily/weekly/monthly snapshot of data.
Usage
Datasets that are created in Insights are automatically visible in source entities within sync studio.
They can be used as source entities, like all other synapse entities. The id field needs to be explicitly defined for a dataset via schema studio. The watermark field is automatically assigned by Syncari (Updated At).
When the pipeline runs, all records from the dataset are read in every sync cycle. Due to this behavior, we recommend setting up your Data Set source with a Sync Schedule.
Dataset used in a pipeline cannot be deleted via insights unless, all dependencies from pipelines are removed. Any edits to the dataset is automatically applied to the dataset used in pipelines.