How to process deleted records for synapses that do not support reading deletes from end systems?
There are many synapses like database synapses (redshift, postgres, mysql), googlesheet, s3, etc that do not support reading deleted records from the end system. To achieve this, below are a couple of recommendations
- Soft deletes - Instead of hard deleting the records in your end system (like database), mark the record as deleted by having a dedicated `deleted` column. Map this field to a field in Syncari. When Syncari reads the record via mapped source, it can read the value of the field and call Delete Syncari Record. At this point you can also call the delete external record to permanently remove this record from the end system.
- If you do not want to have deleted records in the primary table, then maintain a secondary table. When a record is deleted in primary table, make a copy of the deleted record in the secondary table. Use this as a source in Syncari pipeline. When syncari reads a record, it can call Delete Syncari Record. At this point you can also call the delete external record to permanently remove this record from the end system.