Problem

• we have critical files that need an off-site backup
• we want those backups to be encrypted

We could encrypt the files ourselves and rsync the files to another server we use, but there are options that do not require us to have an SSH access to another machine.

rclone

There are many tools to copy files from one server to another. I’ve used rclone a bit recently.

• it’s easy to learn and use
• it targets a lot of storage systems
• it’s safe. Its encryption uses chacha20poly1305 which is the gold standard these days.

Also, it has a nice ande active forum where the project lead offers helpful answers.

Configuration

We’ll need to create 2 remotes:

• one with credentials and settings for the actual “cloud” storage
• one as an encryption proxy

In order to do this, we will need to run rclone config twice. rclone config updates the settings of rclone’s config file in ~/.config/rclone/rclone.conf.

• the first time, we create a remote called my-s3-provider, of type S3. We provide the necessary information and credentials.
• the seconde time, we create a remote called secret, of type crypt. We reference a bucket with the syntax remote:bucket_name, and provide an encryption password

It can look like this (~/.config/rclone/rclone.conf) : we will encrypt things stored in the backups bucket of the my-s3-provider remote.

# ~/.config/rclone/rclone.conf
[my-s3-provider]
type = s3
provider = …
access_key_id = …
secret_access_key = …
…

[secret]
type = crypt
remote = my-s3-provider:backups
filename_encryption = standard
directory_name_encryption = true
password = …
password2 = …

Protecting your credentials

Out of the box, rclone stores the credentials in plaintext in its config file, so anybody with read access to the file will have the credentials. What I find a bit better but still explicit is to store the credentials in an environment variable, and reference it rclone.conf:

secret_access_key = ${YOUR_S3_ACCESS_KEY}

If you have randow go stacktraces when you try to do anything, have a look at your credentials.

Synchronizing backups and download

Ok, so now we can send some files using rclone. We have various options:

• rclone copy copies everything from source to destination
• rclone move copies everything from source to destination, then deletes from source
• rclone sync copies everything from source to destination, then deletes on destination the files that do not exist in the source
• rclone rcat cat be used to stream the content of a file

Let’s create an encrypted-backups at the root of our bucket. Then, we can upload our backups in this directory:

rclone move --progress --s3-chunk-size=20M /your/backup/directory secret:/encrypted-backups --log-file=rclone-upload.log

When you’ll download a file, you’ll do it like this:

rclone copy --max-age 24h --progress secret:/encrypted-backups/ ./backups

When you want to list the content of your directory, you’ll need to use the secret proxy, unless you want the encrypted files:

rclone lsf my-s3-provider:backups/encrypted-backups
** a lot of random gibberish**
rclone lsf secret:/encrypted-backups
** the actual file names**

Going further

There are a lot of other commands, and they have many parameters. Usually, the --help flag can provide a lot of help. Then, knowing some shell script can be useful. Here is how to find the latest file:

rclone lsf --files-only --format "tp" secret:/encrypted-backups | sort | tail -1 | cut -d';' -f2