How does Syncari Handle deleted records?
For deletes made in the end system, syncari records have a boolean flag that should be set to True whenever a record is deleted. These can be set either from the Read or Webhook methods, depending on how the system communicates deletes via the API.
If the end system doesn’t support soft deletes please note that there could occur errors on Update, since the end system record IDs won’t exist anymore. To handle these cases in platform, you can use a custom action to perform a GET request on existing records, to check if they exist or not.
For deletes that happened in other systems and you want them to be propagated to your end system, you need to have the delete method configured and also select the option in the application, per each pipeline, to receive delete requests.
There is also an optional option that allows to recreate the records after they were deleted, which is available on the destination node.
How to handle child/parent relationships?
When it comes to nested objects (for example orders with line items list) there are two approaches to implementing a custom synapse: natively with a child schema or using a multivalue object field.
In the native approach you would set the schema for both entities separately. The child entity would have the child flag set to True on the schema and the parent entity would have a multivalue child type field. For cases where a record has single child records, we recommend flattening them into the parent schema or passing them as an object field, for easier user experience.
Here is an example of the child / parent schema models:
'child': {
'apiName':'child',
'displayName':'Child',
'child' : True,
'attributes':[
{'apiName':'id', 'dataType':'integer', 'displayName':'Id', 'isIdField':True},
{'apiName':'email', 'dataType':'string', 'displayName':'Email'},
{'apiName':'updatedAt', 'dataType':'datetime', 'displayName':'Updated At','isUpdatedAtField':True,'isSystem':True, 'isWatermarkField':True},
]
},
'parent': {
'apiName':'Parent',
'displayName':'Parent',
'attributes':[
{'apiName':'id', 'dataType':'string', 'displayName':'Id', 'isIdField':True},
{'apiName':'name', 'dataType':'string', 'displayName':'Name'},
{'apiName':'utterances', 'dataType':'child', 'displayName':'Child', 'referenceTo' : 'Child', 'isMultiValueField':True},
{'apiName':'updatedAt', 'dataType':'datetime', 'displayName':'Updated At','isUpdatedAtField':True,'isSystem':True, 'isWatermarkField':True},
]
}
This setup will enable the functionality showcased in this article here: https://support.syncari.com/hc/en-us/articles/24313458796308-Working-with-Parent-and-Child-Entities. The native child / parent approach is recommended for use cases where you need to unify child records from different systems and you need to create / update child records on the destination side.
Second approach is using a multivalue object field on the parent record, such as:
{'apiName':'child', 'dataType':'object', 'displayName':'object', 'isMultiValueField':True}
This will create all the child records in the field automatically and won’t offer the functionality to set and convert another system’s child record into our own end system. The multivalue object field approach is recommended for single direction use cases and / or cases where you aren’t unifying the child records and have the system with the child record on the source side. This approach can be also paired with update / create syncari record nodes, if you need to extract the child records as separate Syncari records within the Syncari Platform.
How to handle APIs with no last modified field on records?
If the end system’s API doesn’t provide last modified (watermark field) on the records, but it’s still possible to do either date queries and sort ascending by updated time, the last modified date can be fabricated within the synapse code for each record. In this case the schema of each entity should include a fabricated field with isWatermarkField set to True and setting the current time stamp in epoch milliseconds on the lastModified Record's attribute.
How to handle APIs with no filtering functionality?
For APIs that can’t filter by date, you’d need to pull all the records with each request. In these cases we recommend to use the option on the destination side of pipelines to only push records that were updated within Syncari. More on this here: https://support.syncari.com/hc/en-us/articles/360054608471-Create-Entity-Pipelines#destination-node-configuration
Please note that if the count of records processed on each run will exceed 5000, we recommend adjusting the time of the pipeline to run less frequently. We also recommend separating the entities of Custom synapse in separate pipelines to avoid affecting other systems that don’t have these limitations.
How to handle APIs with no pagination functionality?
We generally recommend as best practice to keep batches of records under 1000 if possible. For APIs that don’t support pagination if the record count won’t exceed 20000, you could process them in a single read method execution, but please keep in mind that this could slow down the pipeline.
If API supports date queries or at least last modified date on the record, those can be used to lower the record count per batch, to implement date filtering within the code and / or to paginate over slices of records within each execution.
Please note that Synapses run on 256 MB of memory while they are executed. If the payload could exceed that the synapse will crash and nothing will be returned. If this could be a scenario for your use case please reach out to our support team.
How to handle APIs with no ID field?
When configuring the source entity schema, you'll usually be working with tables containing proper database IDs; these IDs, in return, will serve as the External ID for any record that Syncari ingests from that source entity. External IDs are central to how Syncari reconciles and deduplicates records shared between systems and so it's essential that the External ID value is immutable and unique.
However, when SQL views, reference tables, and child objects serve as source entities, we may find ourselves without a proper ID field.
Step 1: Add a Fabricated ID Field
Example for Hardcoded Schema:
# main.py
entity_schemas = {
"organizationMember": {
"apiName":"organizationMember",
"displayName":"Organization Member",
"description":"Members belonging to a specific Organization",
"attributes":[
{"apiName":"id", "dataType":"string", "isIdField":True, "isSystem":True...}
{"apiName":"orgId", "dataType":"string", "displayName": "Org ID"}
{"apiName":"email", "dataType":"string", "displayName": "Member Email"}
]
}
}
If you've hardcoded the source entity's schema directly into your main.py file then accounting for this use case is simple - merely add an ID field and mark the isIdField and isSystem flags as True. This field will serve as the fabricated ID field.
Once your custom synapse is deployed, you may choose to create a Syncari entity that mirrors the schema of your source entity, like Organization Member, above. When doing so, the fabricated ID field will serve as the ID field for that newly-created Syncari entity.
Step 2: Identify a Composite or Alternate Identifier
If no explicit identifier exists, we generally recommend creating a composite ID. If the entity you're working with is a child of another entity - for instance, the Organization Member entity has a reference to the parent, Organization entity through the Org ID field - you can create an ID that's a composite of the email address and org ID fields. A composite identifier, in this example, might look something like: 123456|john.doe@example.com, following a structure of:
<parent object/reference field>|<unique identifier>.
Step 3: Create the Composite ID
The creation of the composite ID happens whenever and wherever, in your code, a Syncari Record instance is created. For example, within your custom synapse's Read method, you will have at least one instance of a Syncari Record object being instantiated, like the example below:
# main.py
def read(self, sync_request: SyncRequest) - ReadResponse:
syncari_records = []
entity_name = sync_request.entity.apiName
if entity_name == "organizationMember":
response = self.client.get(request_path, headers=self.headers()).json()
for record in response:
syncari_records.append(Record(id=f"{org_id}|{email}",
name=entity_name,
values=record,
created_at=record["created_at"],
last_modified=record["last_modified"]
return ReadResponse(data=syncari_records,
watermark=sync_request.watermark,
offsetType=OffsetType.NONE)
If you look where the Record object is instantiated you'll see how we construct the composite ID via string interpolation, pulling out both the Org ID and Email field and placing their values on either side of the pipe character. That's it! Once this step is complete, you are ready to begin merging and deduplicating records.