Pipelines enable you to define, manage and control data movement between source nodes and Syncari nodes; and between Syncari nodes and destination Nodes. Pipelines all start with an entity pipeline.
You can define, manage, and control entity and attribute pipelines in various ways in Pipeline Studio. Deciding which method to use depends on your use case. We recommend that you talk with a member of our Onboarding and Support Team about your use case to make sure you are set up for success in reaching your objectives.
Pipeline architecture
In Pipeline Studio, pipelines are organized into two levels: Entity Pipelines and one-level deeper there are Attribute Pipelines for each of your Entity Pipelines. In both pipeline types, here is the anatomy:
Source-side: These are all nodes to the left of the Syncari core node. This is where data enters from your connected source system(s).
Syncari core node: This is where data syncs into Syncari's data store and is hosted for lookups, processes, etc. In an Entity pipeline, clicking the core node gives you access to attribute pipelines and also to Merge Studio. When inside an attribute pipeline, you'll have access to the node configuration and validation.
Destination-side: These are all nodes to the right of the Syncari core node. This is where data flows out to your connected destination system(s).
Order of operations
Syncari processes the data of a pipeline starting from the Source-side → Syncari core node → Destination-side. More specifically, here's the full order of operations:
- Receive Record from Source
- Run Source-Side Entity Pipeline Steps
- Run Source-Side Attribute Pipeline Steps (in no particular order)
- Dedupe and Merge
- Create/Update Syncari Record in Syncari's data store
- Run Destination-Side Entity Pipeline Steps
- Run Destination-Side Attribute Pipeline Steps (in no particular order)
- Create/Update Destination Record
For more in-depth information on each step, go to our How Sync Works? article.
Unfiltered data flow
When Entity pipelines do not have any predicates or decisions (i.e. unfiltered) to both Source and Destination sides of the pipeline, all records that enter from one system will end up being records in all systems.
When Attribute pipelines do not have any decision nodes (i.e. unfiltered) to both Source and Destination sides of the pipeline, the latest value synced from any system will end up being the value in all systems
Pipeline visibility & documentation
Entity Pipelines vs Attribute Pipelines
When deciding between building in Entity Pipelines vs Attribute Pipelines, there are a couple things to consider. Entity Pipelines have a set of functions that can be applied to the entity, while attribute Pipelines have a larger and more granular set of functions that can be applied to the attributes.
Generally, build your nodes and pipeline logic in the Entity Pipeline when possible because it's more discoverable when troubleshooting or determining what the pipeline is doing.
Documentation
It's best to label the nodes in both the Entity and Attribute pipelines to be as descriptive as possible. This makes it much easier for other users to follow along and understand what logic is taking place in the pipeline.
When you have multiple nodes that are all related to each other or a series of nodes contributing to one specific outcome, consider grouping those nodes so that it can be easier to read and the pipeline is more manageable.
Optimizing Pipeline Sync
There are many different approaches you can take when building pipelines, optimizing for sync speed or complexity. Check out our Optimizing Sync article for more details on best practices.
Creating your first pipelines
Consider your use case and select one of these approaches:
- Creating Pipelines Automatically
- Creating Pipelines Manually
- Creating Entities with Pipeline Studio
- Creating Attributes with Pipeline Studio
You can also take a look at our library of Use Cases for additional guidance on Creating Pipelines. (see Use Cases)