Using S3 as a Destination

With syncari it is possible to use an S3 bucket as a destination of a pipeline. The result of mapping an S3 bucket as the destination of a pipeline is that the bucket will receive a new timestamped file with every sync cycle where there is a changed record. It will look like this in the S3 UI

As you can see, when a new record we have never sent before to S3 is being sent it lands in a file with the tag "_create_" in the file name. If we are sending a file that contains updates to records we have already sent it will land in a file with the tag "_update_ in the filename.

Set Up

1. First create a folder that will represent the destination entity in the root bucket mentioned in the S3 Synapse Setup article. In my example I called the folder "S3 Destination Test" and my root folder configured in my synapse is called "onboardingb". NOTE: Destination folders should never be used as a source and vice versa.
2. Create a CSV that has a header row with the columns where you want to receive data. I used Google Sheets to do this. Now upload this CSV to the entity folder in your bucket.
   NOTE: Make sure to keep this file in the folder at all times. Think of this file as the schema of the entity.
   
3. From Syncari navigate to Schema studio.
4. Click the dropdown list and select your S3 Synapse
5. Click the kebab menu on the right hand side
6. Click "Refresh Schema"
7. In a minute the schema will be refreshed and you will see your new entity

Mapping

Now visit the entity where you want S3 as your destination and map the fields as you normally would.

That is it! Once this pipeline gets published, new files will be written to the S3 bucket with every sync.