PieSync Technical Deep-Dive: how does it work?
In this article, we'll take you through some of the technical stuff PieSync does behind screens. Because there's a lot to cover, we've done so in a Q&A format. If you have any technical questions that are now covered here, please let us know!
Q: What is PieSync's sync time from the moment a contact gets updated.
Example: There is an active connection between MS Dynamics 365 and HubSpot CRM. The phone number of a contact that is kept in sync gets updated in MS Dynamics 365. How long will it take for that change to get synced to HubSpot CRM.
A: Within 5 minutes or in real-time. Every 5 minutes, PieSync will ask your connected apps for a list of contacts that have been updated since the last time it asked. Then it will check each contact individually to see what was updated. If it is a change that should be synced, PieSync will update the other app. For some apps we support, we use webhooks (eg. Mailchimp), this allows us to sync these kinds of updates in real-time.
Q: How many API calls does a connection/sync consume?
A: This depends on (1) whether the connection is doing an initial or incremental sync and (2) which specific connectors (read: applications) are involved.
The initial sync is the state of a connection when it has been activated for the very first time or when there's been a reconfiguration. An incremental sync is the state of the connection when it active and syncing new updates.
- In the case of an incremental sync, the number of API calls is smaller than for an initial sync. What happens during an incremental sync:
- PieSync will do API calls (list) to App B to index creations/updates in App B that the sync engine has not yet detected.This would mostly fit in a single API call to B.
- The engine then fetches N changed contacts (or companies, leads, …) from side A. This would, most of the time, fit in a single API call to A.
- Then, the engine would issue at most N find calls to B, for the contacts from the set of N contacts that are already linked or are matched with a contact from B.
- After merging all contacts, we would execute at most N contact creation or modification calls in B and at most N modification calls in A.
- Possibly, nothing happens if the contacts are not in scope according to the customer's rules, or nothing has to actually change in either app (depending on rules/mappings).
The process of an initial sync is similar, however, N will be much larger, requiring more API calls:
- PieSync will start with indexing all contacts in App B. The amount of calls depends on the page size of B. Assuming this is 100, this would take B/100 api calls.
- Now PieSync will sync from A to B (and index A at the same time). Similarly, they'll fetch all contacts from A. Each contact will either have no effect in B (not in scope or no change needed), or result in a creation/modification in B (whether a matching contact was found or not). So they will have an amount of find calls + an amount of update/create calls which heavily depends on (1) customer rules/mappings (2) match rate (3) batching possible or not. Possibly, they’ll also do update calls to A at the same time (it’s quite possible that merging two contacts results in updates on both sides).
- Lastly, PieSync will sync from B to A. Here, we’re fetching all contacts again. The contacts that were already processed in the previous step will be skipped. Other contacts will either have no effect (out of rule scope) or will result in creations in A.
In general, the amount of calls is much less when batching is supported in a connector, that will allow PieSync to execute multiple finds/updates/creates in a single API call.
- This process is a bit different if a connector purely relies on webhooks, without having a way to fetch changed contacts since a certain time. However, this is a small minority of use cases.
Q: What happens in case of errors while syncing data?
A: When one or more contacts fail to sync, they will be listed per failure reason in PieSync Health.
If a contact fails to sync, it mostly means that the receiving app refused PieSync to create or update them. This can be due to some requirements, technical misalignmets, or due to PieSync exceeding the API call limit.
In case PieSync exceeded the API call limit, then PieSync will automatically try to sync again once the applicable time period has passed.
In the event of other errors with syncs, Health should provide more information about the error that can be used to resolve it.
Q: Do empty values ever overwrite existing values?
A: Not during the initial sync. Then PieSync will only enrich data, not overwrite existing values. In other words, the integration will find any gaps on either side of the two-way sync and fill those gaps where possible. This is what's called a "smart merge."
This is not the case during incremental syncs. Once the sync is up and running and you delete a value, PieSync will consider this to be an update that needs to be synced like any other. So in that case an empty value will overwrite an existing value.
Q: Is there an option to update a property only if it's blank in one app A, but not if populated in both?
A: Yes, but not by default. PieSync's default behavior is the "smart merge," filling in existing blanks if data is known in one system. In a two-way sync, PieSync will simply propagate changes on either side.
If you want more customization, when configuring the field mappings, you can make a rule for one-way mappings only or disable certain mappings altogether.
Q: Is data encrypted in transit?
A: The platform that externally transfers customers’ data on their behalf to and from upstream application APIs is encrypted in transit.