Create HTTP Synapse
To create an HTTP synapse, go to the Custom Synapse tab in Synapse Studio. Click on the "Create New" dropdown and select "HTTP Synapse."
Enter the Display name and API name for the synapse and choose the Authentication type.
We support three authentication types
-
API Key: Requires an API key for authentication.
-
OAuth with Client Credentials: Requires a Client ID, Client Secret, and Endpoint URL for authentication.
-
Bearer Token: Requires a Bearer Token for authentication.
Optionally, you can upload an icon for the synapse and add additional metadata.
Click "Next" to proceed to the Test & Authenticate step. Here, you can test your HTTP endpoint with the selected Authentication type. Select the HTTP method, enter the endpoint, enter the request body and headers, and test. You can view the response preview in the section below.
Click "Next" to move to the review step and review your configurations. Click "Create," and your HTTP synapse will be created in the draft state.
Create Entity
To create an entity, select the "Edit Entity Schema" option in the HTTP Synapse.
It will take you to the entities page and click on the "New Entity" button to open the entity creation wizard
In the Basic settings step, enter the Display name and API name for the entity. Optionally, you can add a description and tags.
Click "Next" to proceed to the Configuration step. Here, you can provide the endpoint, headers, body, variables, schema, record fields, and pagination type based on your requirements to configure your entity. The authentication type defined in the HTTP synapse will be used to authenticate the endpoint.
Once you have entered your endpoint URL and optionally configured headers, body, and variables, go to the Testing tab to test your HTTP endpoint configuration. After clicking the Test button, the endpoint will be called with your configuration, and the response will be displayed below. You can click the "Use Response as Schema" button to generate a schema based on the endpoint response.
Next, go to the Records tab and enter the record information for your response.
Record selector: Specify an XPath expression to extract records from an array within the response. If left blank, the entire response will be treated as a single record, unless the root of the response is an array.
ID selector: Enter an XPath expression to specify the field used as the unique identifier (id) when extracting records. Leaving it blank will generate a hash of each record as its identifier. The id field is relative to the record selector.
Watermark Selector: Specify an XPath expression to identify the watermark field used for extracting records. If left blank, the API calling time will be used as the watermark. The watermark field is relative to the record selector.
Eg. Let’s take the following sample response:
{ "response": [ { "id": 1, "name": "Name 1", "age": 21, "address": { "street": "1 Example St", "city": "City 1", "state": "State 2", "country": "Country 2" }, "updatedAt": "2024-10-10T09:39:19+05:30" }, { "id": 2, "name": "Name 1", "age": 21, "address": { "street": "1 Example St", "city": "City 1", "state": "State 2", "country": "Country 2" }, "updatedAt": "2024-10-10T09:39:19+05:30" } ] }
This payload contains an array called response with two records. Each record represents a person with attributes like id, name, age, address, and updatedAt.
Here’s how you would configure this payload:
-
Record Selector: Use
"response"
as the record selector because it identifies the array within the payload that holds the individual records. - ID Selector: Use
"id"
as the ID selector since theid
field is unique for each record. - Watermark Selector: Use
"updatedAt"
as the watermark selector.
If your response is paginated, configure the pagination options in the next step.
Note: If you're using cursor pagination and rely on a next cursor selector to extract the cursor from the response, an XPath expression can be used. In cases where the response does not provide a dedicated cursor field, and the cursor needs to be determined as the id
of the last record (as shown in the example JSON), the XPath expression /response[last()]/id
can be used as the next cursor selector.
Click "Next" to move to the Review step, where you can review your configurations. Click "Save" to finalize and create your entity.
Once you are done with the configuration, you can publish the HTTP synapse by clicking the "Publish" button in the top bar.
Activate HTTP Synapse
HTTP synapses will be available along with other synapses in the right-side panel. Drag and drop the synapse onto the canvas and configure it by giving it a name and selecting the Authentication method. Click "Next." In the next step, authenticate your synapse by providing the credentials. Click "Next" and activate your synapse.
Create entity pipeline and Sync
Before creating the entity pipeline, check if your schema is configured correctly by going to the Schema Studio.
Next, go to the Sync Studio and create the entity pipeline for your HTTP synapse. Go to the draft entity, configure the pipeline based on your requirements and publish the pipeline. After that, click on the "More Actions" dropdown and perform a Resync. The entity pipeline will change to the running state after some time.
Now you can go to the Data Studio and see that the entity you created now has the data returned from the API endpoint.