Client-side encryption with own key?

[bitwombat]

AWS cli supports server-side encryption with a client provided key via options --sse-c and --sse-c-key.

s3cmd seems to support 'client-side encryption' with -e (not sure why the nomenclature difference), and server side with AWS storing the key.

Any plans for s3cmd to support encryption via a key only I know, like --sse-c with AWS cli does?

[foxx]

This already seems to be supported. You have to set your GPG passphrase using s3cmd --configure

[bitwombat]

Good spot @foxx! For the record, s3cmd uses gpg, but won't use it for sync:

if cfg.encrypt:
        error(u"S3cmd 'sync' doesn't yet support GPG encryption, sorry.")
        error(u"Either use unconditional 's3cmd put --recursive'")
        error(u"or disable encryption with --no-encrypt parameter.")
sys.exit(EX_USAGE)

But it does support it for put. To enable it, have gpg available on your system and set up ~/.s3cfg as follows:

encrypt = True
gpg_passphrase = somethingtricky

These were set for me, presumably by s3cmd --configure originally:

gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s

Oddly, during configuration, when s3cmd prompts for a gpg password, it says:

Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3

But this isn't SSL we're talking about. It client-side encrypts each file with gpg and your supplied password, and they get stored that way on S3's server. So what that says is true, but it's not just protected "while in transfer".

This feature keeps everything in your control - the key (well, symmetric encryption password) is under your control, the file leaves your system encrypted and is stored that way. Beaut!

[foxx]

Yeah to be honest I've had similar issues using sync for client side encryption. You could use encfs+rsync but it's hacky and has some pretty serious security flaws.

Eventually I conceded and wrote my own tool for performing a client side encrypted s3 sync, with filename encryption, file size obfuscation, optional copy/move support (to prevent re-uploading on rename) and path flattening (so structure cannot be determined). I'm building the first prototype in Python for rapidly creating acceptance tests, and then it will be re-written in C.

It will be a month or so until it's ready for public consumption, but feel free to watch https://github.com/foxx/encsync

This tool is being written for production usage so, despite the lack of maturity, you can take some comfort that the first release will have been battle tested before being opened to the public

[bitwombat]

Will be watching encsync!

The problem with encfs+rsync is you can't use include/exclude filters.

I ended up with a home-rolled solution where I:

1. copy my tree to be backed up using hard links (e.g. cp -l src mirror).
2. run rsync --ignore-existing --ignore-non-existing src/ mirror/ so that it just deletes all the files I want excluded.
3. encfs --reverse
4. run rsync to upload everything to my untrusted remote storage.

This accomplishes a few things:

1. no extra disk space req'd
2. leaves my system encrypted
3. fast
4. filterable - using the same exclude patterns I use for local external disk backups
5. incremental transfers work

I know we're warned off encfs by the experts, but I'm not a bank - I just need to keep mostly casual people out of my stuff. The --reverse feature is irresistible.

It took me weeks to come up with this scheme after looking at a lot of existing solutions.

[foxx]

Sadly encfs wasn't an option for me because it has several shortcomings;

1. Poor encryption implementation
2. Meta leakage (directory structure and file sizes)
3. Requires a user space driver

However if you're willing to accept those shortcomings, then it's probably the cleanest solution out there right now (until encsync is finished!).

[mdomsch]

I haven't worried about extending s3cmd's GPG implementation. I much
prefer to see a tool (of your choice) used to encrypt any content needed,
and then invoke s3cmd to transfer it to S3. Likewise on receive +
decrypt. Yes, it requires you to have enough local disk space to store the
encrypted copy.

On Mon, Sep 5, 2016 at 7:04 AM, Cal Leeming notifications@github.com
wrote:

Sadly encfs wasn't an option for me because it has several shortcomings;

1. Poor encryption implementation
2. Meta leakage (directory structure and file sizes)
3. Requires a user space driver

However if you're willing to accept those shortcomings, then it's probably
the cleanest solution out there right now (until encsync is finished!).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#774 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAqDqmrdUFPgJTYZREhCCy3AY-vcXJUWks5qnAU9gaJpZM4JnY0L
.

[foxx]

"I haven't worried about extending s3cmd's GPG implementation"

That's because s3cmd wasn't designed for data privacy in mind, when using sync for backups

"Yes, it requires you to have enough local disk space to store the encrypted copy"

That's a problem, especially when you are doing 4 concurrent uploads of 20GB+ files, not to mention a waste of disk space, performance and disk life. Streaming would be a much better option here.

[fviard]

@bitwombat @foxx If you have some knowledge of python, don't hesitate to have a look at the inner logic of s3cmd and so why the sync could logically not work with the gpg mode as is. Maybe you can be able to create a PR with ways to overcome the issues. But it is a really complex problem.

One major rework that I have in mind but that is not trivial to implement, is to add the feature to be able to have a local "sqlite" db cache. In it, when using in linux, you can keep something like the ctime of a file, with its hash (md5 or sha256), thus, you will not be needed anymore to recalculate all the times the local files to see if there was a change. Also, this could help limit the memory usage of s3cmd for big datasets. And in the end, this could possibly keep what is needed hash of the encrypted version of a file to do a sync.

[foxx]

@fviard I have already got a working Python prototype of client side encrypted sync, complete with backup and restore. You are right, it's a complex problem indeed, and you are more than welcome to copy the code from my repo to use in s3cmd if you wish. It will be a few weeks until I'm ready to make it public. The prototype for building acceptance tests is using Python, then it'll be ported to C.

" to add the feature to be able to have a local "sqlite" db cache"

This is really not a good idea, several other products use this approach but the database quickly becomes corrupt or out of sync far too easily. Instead, it's better to keep an encrypted copy of the metadata in the filename itself. The path also needs to be flattened, otherwise directory structure and file sizes are leaked, which can be used to make assumptions on what the content might be.

Furthermore, file sizes need to be obfuscated by rounding them to a incremental block size. This is next on my todo list, I want to see if GPG has the ability to set block size, if not then I'd need to wrap the file with tar, which I really don't want to do.