What is a Bi-Directional Sync and how does it work?

September 27, 2016
All posts

Bi-Directional Sync and APIs

A Bi-Directional sync works by gathering and sharing data from two different systems through their application programming interfaces (APIs).

In our cloud-based business world, we usually call systems “apps”, examples of apps can be HubSpot or MailChimp.

Every app has an API which acts as a gateway (the technical term for gateways is “servers”), through which data can be extracted, or “pulled”.

In order to pull this data from the app and share it with a different app, you need a middleman, or “client”, such as PieSync.

The client pulls that data from one API and then transforms and submits this data to the second API.

What’s an API?

APIs are a way for computers to talk to one another in a way that they can quickly and easily process information. Each API has a unique “protocol”, which is like a special computer language.

The key thing to note is that APIs can’t talk to one another directly. Not only because they use different protocols, but because they are dormant until activated by a request for information.

PieSync acts as the middleman to request the information. PieSync then translates the information so that the other API can understand it.

The problem is, there’re not just one or two protocols floating around. Every developer can develop a new protocol, and this was the case several years ago. Over time, some standards have started to emerge, with protocols like REST and SOAP being popular.

Let’s look at an example, say we have an app named “EmailMarketing” which uses the protocol SOAP, and another app named “EzyCRM” that uses the protocol REST. We want to sync data between these two apps.

In order for EmailMarketing to tell EzyCRM that there is an update on a contact name, you need a middleman, PieSync, to translate the protocol and hand over the data.

You’d think that would be the end of it. But it’s not. Not only does PieSync “translate” the protocols, it also has to manipulate the structure of the data so that the second app can understand how to structure the data it is receiving. PieSync’s CTO Mattias Putman elaborates:

“A protocol is a way of communicating data, but it doesn’t say anything about the structure of that data, or the “data model”. The “data model” is defined by the specific API itself, and not by the protocol. For example, some APIs represent emails as lists of an email value associated with a label. Other APIs represent an email with just a single email field.”

The PieSync connector pulls the data from EmailMarketing’s SOAP API and then hands this information to our platform.

Our platform will then look at the data structure and match it up to EzyCRM’s data structure.

Our platform then hands this information to EzyCRM via the PieSync connector that communicates in the JSON/REST protocol (JSON is a way of encoding data and is mostly used together with REST).

“We talk in SOAP terms with EmailMarketing to get contact data out,” says Putman “This contact data is modeled completely different in EzyCRM, so we have to translate the data to EzyCRMs data model. Then we talk in JSON/REST terms with EzyCRMs API to send the translated data over.”

Smart Syncing

PieSync does more than just translating protocols and manipulating data structure. PieSync is also about “smart syncing”.

Basically, PieSync tracks every mapped field between the two connected apps to see if there are any changes. If there are changes in one app, PieSync will pull this new data, translate the protocol, manipulate the data structure, and push the change into the second app.

If there are no changes in either app, PieSync does nothing.

When we’re scanning for changes, we take fields that are the common denominator between any the apps, and sync those.

“If app A has emails and name, app B has name, and app C has emails. We sync the common denominator between each combination of two apps out of A B and C. We always do the most optimal mapping between any pair of apps,” says Putman.

If PieSync checks the two apps through their APIs, and sees that there are changes on both sides to the same field, it looks to see which app you have configured to “win” in this situation, PieSync then makes the change in favor of the winning app. This ensures that data isn’t overwritten unintentionally.

There are exceptions to the “win” rule, for example if both fields you want to sync are list fields. PieSync can merge all the items without losing any data. This is another reason PieSync is about smart syncing, we’re able to interpret the structure of the data.

How does PieSync know that the data changed?

There are a few ways that PieSync knows data has changed. I will cover the two most popular ones:

Webhooks - PieSync says to app “If you have a change in a record, please submit the fact that it changed to this URL we provide.” In other words, we ask them to send us the information that something has changed. This is the most efficient way, however a lot of APIs don’t support it because it is complicated to implement reliably.

Polling Frameworks - This is where PieSync asks every five minutes - “Hey did anything change? If so give me all that changes that happened in the past five minutes.” This is the one most connectors use.

Maintenance Nightmare

Maintenance of these API connections is tough. Developers upgrade APIs, and old APIs fall out of use, or “deprecate”.

This means that our developers have to continually ensure that the connections to the APIs are up to date, so that PieSync can still pull and submit data.

If APIs are upgraded or deprecate, our developers then have to re-write our side of things to match up to the upgrades or to the new version of the API.

Most developers from these apps notify us either directly or through developer blogs, about these changes. But sometimes we only find out when our connection to their API is broken. Our developers then drop everything to fix this connection.

Putman is an advocate for SaaS companies investing in Developer Experience stating:

“Next to heavily investing in User Experience, Product companies should equally invest in Developer Experience. This is of critical importance to get developers on board with integrating your product and API’s into the massively growing SaaS market”

Is PieSync the same as Zapier?

The short answer: No.

Zapier is different from PieSync, in that it is a task runner that transforms data.

“A big difference is that we understand data and Zapier doesn’t. They just do dump data copying. We understand how your data is modeled in every app and therefore we can do more advanced transformations and do so more reliably,” says Putman.

For example: someone fills out a webform on your landing page, Zapier will take this information, and transform it into information that makes sense to the receiving app, such as a CRM. However it is up to the user to make the decision if this transformation makes sense, Zapier doesn’t help you in any way, and it doesn’t look at the data models.

Zapier can also unintentionally overwrite information in the receiving app, because it doesn’t look to see if the information is the same.

It doesn’t perform syncing operations because quite often it doesn’t make sense to do this between the apps they connect. There is no point syncing your webform and your CRM, they have entirely different functions from one another.

Zapier just does what it’s told, while PieSync pulls the data, and sees if any action is necessary, and it only syncs if it needs to.

As you can see what we do here at PieSync is enormously complex. Very few dare to take this task on, simply because the sheer amount of work involved is formidable.

Want to give Bi-Directional syncing a try? Sign-up today for a 14-day free trial.

Start my Bi-directional Sync

About Vanessa Rombaut
Vanessa is a wordsmith extraordinaire. Originally from Australia, she has travelled the world and the seven seas to write scintillating content for you to enjoy. She likes books, travel, vintage films and sushi (not necessarily in that order).