Pipelines enable you to define, manage and control data movement between source nodes and Syncari nodes; and between Syncari nodes and destination Nodes. Pipelines all start with an entity pipeline.

You can define, manage, and control entity and field pipelines in various ways in Sync Studio. Deciding which method to use depends on your use case. We recommend that you talk with a member of our Onboarding and Support Team about your use case to make sure you are set up for success in reaching your objectives.

Pipeline best practices

Pipeline architecture

In Sync Studio, you have Entity Pipelines and one-level deeper there are Field Pipelines for each of your Entity Pipelines. In both pipeline types, here is the anatomy...

1. Source-side: this is all nodes to the left of the Syncari core node and includes the data source(s) that you want to bring into Syncari.
2. Syncari core node: This is where data syncs into Syncari's data store and is hosted for lookups, etc. When working with the Entity pipeline, you can click the core node to get to the field pipelines and also access the Merge studio. When inside a field pipeline, you'll have access to the node configuration and validation.
3. Destination-side: this is all nodes to the right of the Syncari core node and may include the destination(s) that you want data syncing out to.

Order of operations

Syncari processes the data of a pipeline starting from the Source-side → then the Syncari core node → then the Destination-side. More specifically, here's the full order of operations...

1. Receive Record from Source
2. Run Source-Side Entity Pipeline Steps
3. Run Source-Side Field Pipeline Steps (in no particular order)
4. Dedupe and Merge
5. Create/Update Syncari Record in Syncari's data store
6. Run Destination-Side Entity Pipeline Steps
7. Run Destination-Side Field Pipeline Steps (in no particular order)
8. Create/Update Destination Record

For more in-depth information on each step, go to our How Sync Works? article.

Unfiltered data flow

• When Entity pipelines do not have any predicates or decisions (i.e. unfiltered) to both Source and Destination sides of the pipeline, all records that enter from one system will end up being records in all systems.
• When Field pipelines do not have any decision nodes (i.e. unfiltered) to both Source and Destination sides of the pipeline, the latest value synced from any system will end up being the value in all systems

Pipeline visibility & documentation

Entity Pipelines vs Field Pipelines

• When deciding between building in Entity Pipelines vs Field Pipelines, there are a couple things to consider. Entity Pipelines have a set of functions that can be applied to the entity, while Field Pipelines have a larger and more granular set of functions that can be applied to the fields.
• In general, it's better to build your nodes and pipeline logic in the Entity Pipeline when possible because it's more discoverable when troubleshooting or determining what the pipeline is doing.

Documentation

• It's best to label the nodes in both the Entity and Field pipelines to be as descriptive as possible. This makes it much easier for other users to follow along and understand what logic is taking place in the pipeline.
• When you have multiple nodes that are all related to each other or a series of nodes contributing to one specific outcome, consider grouping those nodes so that it can be easier to read and the pipeline is more manageable.

Optimizing Pipeline Sync

There are many different approaches you can take when building pipelines, optimizing for sync speed or complexity. Check out our Optimizing Sync article for more details on best practices.

Creating your first pipelines

Consider your use case and select one of these approaches:

• Creating Pipelines Automatically
• Creating Pipelines Manually
• Creating Entities with Sync Studio
• Creating Fields with Sync Studio

You can also take a look at our library of Use Cases for additional guidance on Creating Pipelines. (see Use Cases)