Syncari pipelines are asynchronous by default. They run at a set cadence (once every minute) and poll the source systems for delta changes. This works well for most use cases, however there may be a need to run the data flow pipeline synchronously to support realtime operations. We have introduced realtime ability on Syncari pipelines to achieve this.
Enabling Realtime
By enabling this flag, the pipeline turns into a realtime pipeline. It generates a URL that can be called by the clients to post JSON data. Choose a webhook source that can receive data. If you have not created one already, you can do so by following this article. The client can use Syncari api user client id/ client secret for authentication.
Authentication
To send an authenticated request to the realtime pipeline, create a Syncari api user in Settings -> Users. This user would have Client Id and Client Secret. Get the access token as described here.
Handling Response
Once the realtime pipeline flag is turned ON, you can now build out the pipeline as needed. The pipeline can be a combination of synchronous and asynchronous. To terminate the synchronous path and return the response to the caller, use the newly introduced Respond to Webhook action. This will allow the synchronous api to be completed while still running the asynchronous parts of the pipeline offline.
Validations
- The realtime pipeline can have only one webhook source. No other source types are allowed other than webhook synapse source.
- The Respond to Webhook node cannot have multiple inbound edges.
- The requests sent by the client must have the header `syncariid` with the instance id as value passed
Testing
The realtime pipeline can be tested using the webhook payload under Run Live Tests action. The response will be logged in pipeline logs. For the API key use the client secret generated for the synacri api user.
Publish
Once the pipeline is validated and tested, go ahead and publish. The realtime flag cannot be edited on a published pipeline. You can however fetch the URL using the settings gear.