As users start building pipelines in Syncari, we work with draft pipelines which are not live yet. The recommended best practice is to build the pipeline in draft, test the pipeline with sample datasets, iterate and modify the pipeline as needed and retest, once the pipeline is executing as expected, go live by publishing it. During the testing phase, in-order to troubleshoot and identify the issue, Syncari launched pipeline log capability where users can access the input/outputs of each node and any errors that occurred. In this article we'll walk through how it works. This feature is available on published pipelines as well.
The pipeline logging capability is available as a setting on each pipeline. To turn ON this feature, navigate to More Actions on the pipeline, click Settings and turn ON the flag for Enable Node Logging, and click Save.
Once it is enabled, rerun the test on the pipeline. Once the pipeline test completes, you can see the results of each node execution in the Logs tab.
It has detailed information on what was the input/output for each node. If there were errors executing the node, the error message and details will be shown in the table. All function nodes and action nodes will be captured in the log for each record passing through the pipeline.
NOTE
To enable logging for an already published pipeline, create a draft of the pipeline, enable the feature from settings and publish the pipeline.
NOTE
While most columns are self-explanatory, Batch Mode is new. Some
actions like 'Create External Record' collect incoming records into
batches and execute the batch. Logs will be marked as
BUFFER_RECORD
for such actions when they are collecting records. When the batch
is executed, you will see another entry for the same action in
EXECUTE_BATCH
mode. Most actions and functions are not batched, and will show
UNBATCHED
as the batch mode.