S3 object storage as backup or cache for interface projects

A practical example of how you can use S3 storage as a backup for PDF invoice documents in a GDPR-compliant way.

In this article we want to show on a practical example how to dock Synesty to an S3 storage to store file backups at an interface project / integration.

We focus on S3 storage solutions that explicitly offer data storage in Germany. This is important in the context of Synesty for data protection reasons / DSGVO, because many customers process personal data (so-called PII Data), such as orders from online stores and marketplaces.

What is S3 Object Storage?

S3 was originally created by Amazon AWS and provides an application programming interface, the S3 API, that allows applications to store and download data to an S3 object store. It is called an object store because in addition to the actual file content, a unique identifier (e.g., a UUID) and individual metadata can also be stored. One difference of object stores in contrast to file stores (e.g. via FTP) is that there are no hierarchical folder structures. Instead, it is a flat structure that only needs one unique identifier per object (file).

Advantages of S3 storage solutions

Security: Data transfer via SSL / HTTPS and often also optional encryption at the S3 provider (Data at Rest). Compared to the insecure FTP protocol a big plus point
often very cost-effective for large amounts of data
Creation of temporary URLs / links that expire after a certain time. This is supported by the S3Upload Step.
Lifecycle features: S3 services often offer to create rules like automatic deletion of all data older than X days / hours.
File versioning: some S3 stores support features where every change to an uploaded file is kept. This makes it possible to keep a change history of files
Possibility of hosting static web pages and files

What is external S3 storage suitable for in the context of interfaces, integration and automation?

If you build automated processes with Synesty, then files are often exchanged with other systems. For this to work, files have to be stored somewhere external (i.e. outside of Synesty) so that Synesty or another system can access them. Classically, an FTP server or the secure variant SFTP or FTPS is often used for this purpose. Or other customers rely on services like Google Drive, Dropbox or Microsoft OneDrive. S3 is another option for storing large amounts of data on a file basis.

S3 storage works well as additional backup storage to which you copy all files processed in flows, so you have a history of those files at all times. Some examples are:

backup of invoices or PDF documents from store systems
caching temporary files of a flow execution e.g. for troubleshooting / debugging over a certain period of time
sending files via time-limited 1-time link
keeping product data from suppliers e.g. to prove errors (example: supplier sends stock or price of 0, but claims the opposite)
In the e-commerce context, the S3 hosting capability is useful to store data feeds with product data for price portals or ads campaigns in a publicly accessible way so that they can be downloaded externally

Practical application using Gridscale.io as an example.

In the meantime, there are several other hosting providers that also have S3 storage in their program and offer an S3 API-compatible interface.

The advantage: Storage solutions that are compatible with the S3-API can be accessed in the same way. This also makes the Synesty Add-On work with many other providers, even though it was originally developed for Amazon S3.

In this article we will use the example of the Cologne provider gridscale.io to show how Synesty can be used to store data on S3 storage. The example scenario is a backup of PDF invoices to S3 storage, which we download from the service Getmyinvoices via Add-On.

The final result looks like this:

A flow,
which downloads invoice documents from October
from Getmyinvoices
and uploads them back to S3.

Account anlegen

Create account

To do this, we register a free account at gridscale.io. We can then find the S3 storage under Storage / Object Storage.

We create a bucket (that’s like a big folder for our files) which we call s3-demo.

After that we create an access key. These are the credentials we need to access this bucket with the API. We get an access-key and a secret-key.

Set up Synesty

Back in Synesty Studio, we create a test flow. This test flow creates a simple test file (with the StringToFile step) and uploads it to S3 with the S3Upload step. For this we enter our previously fetched access data by creating an account.

Additionally, we specify the endpoint there. This is the important part: With this we determine to which service (API) we actually connect. For the gridscale servers in Frankfurt, this is https://gos3.io, which you can read here. The connection test says that the connection could be established successfully and that we have access to the s3-demo bucket.

With this, our test flow is almost ready. We still specify the bucket (s3-demo) where we want to upload our test file and press execute flow.

In the S3 hump at gridscale we see the uploaded test file.

Two options of the S3Upload step should also be discussed here:

makeFileDownloadable: If this option is set to “Yes” and the file is downloaded from S3 using a browser, the browser will display a download dialog instead of opening the file directly in the browser.
presignedUrlValidityInMinutes: This allows us to create a timed link of the file.

Especially the option presignedUrlValidityInMinutes is interesting. For example, with this we can send someone the links to the uploaded files, but this link will expire after 5 minutes. This is very valuable from a security point of view, as you can easily share sensitive data with others, but without the hassle of setting up credentials. The link is allows access only for a short time window. After that, access is no longer possible.

Now that we have successfully tested the transfer, we can rebuild the test flow to back up our PDF invoices from Getmyinvoices to S3 storage.

Template to try out

The examples from the article can be installed as a template, tried out and adapted for your own purposes.

Some more S3 providers with data center in Germany

Flying Circus
Timme Hosting Object Storage
IONOS S3 Object Storage
Strato HiDrive S3
Plus Server
Telekom Cloud Object Storage Service (OBS)
Dunkel Cloud Storage
OVH Cloud Object Storage
somewhat more technical: s3proxy A self-hosting solution to address an S3-compatible API endpoint behind which you can put any other non-S3 storage solutions (such as Google Cloud, Microsoft Azure, OpenStack).

This is a small selection, not exhaustive. There are other hosting providers that also offer S3 object storage, or allow you to build your own object storage via e.g. Nextcloud or Owncloud.