on btrfs, gocryptfs is significantly slower than encfs

[tyrak]

Hi,

I just discovered this project and I am considering to use it to replace encfs, but it's write performance is significantly worse than encfs on my laptop. This is a sandy bridge laptop with a 5400 rpm (spinning) hard drive running 64-bit linux. I tried the provided precompiled binary for debian as well as compiling it myself (both with and without openssl support) and the results are very similar.

Here are some benchmarks:

$ ./benchmark.bash 
Testing gocryptfs at /tmp/benchmark.bash.5r4.mnt
WRITE: 131072000 bytes (131 MB) copied, 89.1114 s, 1.5 MB/s
UNTAR: 914.49
LS:    2.05
RM:    9.34

and

$ ./benchmark.bash -encfs
Testing EncFS at /tmp/benchmark.bash.QZU.mnt
WRITE: 131072000 bytes (131 MB) copied, 3.32415 s, 39.4 MB/s
UNTAR: 20.68
LS:    3.54
RM:    4.99

[rfjakob]

Hi tyrak, thanks for the report! I definitely want gocryptfs to be fast on hdds as well. I'll go find one and get back to you late next week, as I'm abroad right now. Gocryptfs uses 4kb blocks for writing and I expected the kernel to merge them to bigger blocks before it hits the disk, but that's probably not good enough on hdds.

[tyrak]

Thanks for the quick reply. I did some further tests. As it turns out, the culprit is not the fact that I am using a (spinning) hard drive, but rather the file system I am using, namely btrfs. I ran the exact same tests as above, on a ext4 volume this time, in the same disk as before. Here are the results:

$ ./benchmark.bash
Testing gocryptfs at /mnt/ext4/benchmark.bash.jSH.mnt
WRITE: 131072000 bytes (131 MB) copied, 1.01945 s, 129 MB/s
UNTAR: 12.59
LS:    1.24
RM:    4.48

$ ./benchmark.bash -encfs
Testing EncFS at /mnt/ext4/benchmark.bash.Kqv.mnt
WRITE: 131072000 bytes (131 MB) copied, 1.65637 s, 79.1 MB/s
UNTAR: 14.90
LS:    3.89
RM:    3.37

As you can see, gocryptfs is substantially faster than encfs in this case. So there is an issue with btrfs. Maybe you are using fsync excessively or some other operation that is slow under btrfs?

[rfjakob]

@tyrak I have pushed a change that coalesces small writes into bigger blocks now. If you get to it, please check out the "btrfs" branch and post the benchmark results. BTW benchmark.bash now takes a path argument :)

[rfjakob]

My results: https://gist.github.com/rfjakob/0d74efdfb83c07db24635da25f289536

[tyrak]

Thanks for the patch. I tried it and the numbers have improved, but the UNTAR benchmark is still slow:

$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.NGy
WRITE: 131072000 bytes (131 MB) copied, 7.05766 s, 18.6 MB/s
UNTAR: 454.42
LS:    1.29
RM:    10.73

[rfjakob]

What gocrypts is doing differently than EncFS is that it preallocates the space before writing a block. This makes sure it does not run out of space in the middle of a write. I have timed how long that takes on ext4 and btrfs during the tar extract benchmark:

ext4:

Prealloc: 18.754µs, Write: 7.532µs
Prealloc: 24.04µs, Write: 9.322µs
Prealloc: 8.466µs, Write: 8.58µs
Prealloc: 9.16µs, Write: 4.739µs
Prealloc: 7.912µs, Write: 9.016µs
Prealloc: 9.738µs, Write: 8.068µs
Prealloc: 8.781µs, Write: 7.374µs
Prealloc: 8.044µs, Write: 8.27µs
Prealloc: 10.62µs, Write: 11.295µs
Prealloc: 8.545µs, Write: 7.474µs
Prealloc: 9.952µs, Write: 10.287µs
Prealloc: 8.943µs, Write: 7.319µs
[...]

btrfs generally looks like this:

Prealloc: 204.07µs, Write: 10.428µs
Prealloc: 189.509µs, Write: 11.735µs
Prealloc: 1.148818ms, Write: 15.123µs
Prealloc: 6.53432ms, Write: 11.962µs
Prealloc: 224.249µs, Write: 12.288µs
Prealloc: 402.438µs, Write: 15.157µs
Prealloc: 265.148µs, Write: 14.551µs
Prealloc: 328.26µs, Write: 11.468µs
Prealloc: 290.392µs, Write: 16.553µs
Prealloc: 233.991µs, Write: 10.779µs
Prealloc: 177.823µs, Write: 12.756µs
Prealloc: 1.158379ms, Write: 10.462µs
Prealloc: 298.516µs, Write: 16.475µs
Prealloc: 259.497µs, Write: 15.527µs
Prealloc: 1.818556ms, Write: 45.656µs
Prealloc: 3.098858ms, Write: 17.108µs
Prealloc: 307.257µs, Write: 18.329µs
Prealloc: 233.511µs, Write: 10.417µs
[...]

and every now and then I see sequences of this:

Prealloc: 34.118642ms, Write: 89.893µs
Prealloc: 46.081966ms, Write: 18.067µs
Prealloc: 60.740359ms, Write: 16.502µs
Prealloc: 48.189781ms, Write: 20.414µs

In summary, preallocation on btrfs is much slower than on ext4. And this is the reason gocryptfs is slow on btrfs.

[rfjakob]

@tyrak I have just pushed another commit to the btrfs branch. It adds the -noprealloc option, which disables preallocation. When this is on, the last 4kB block of a file may become unreadable when you run out of space in the middle of a write. But this can happen on EncFS as well, and I have never seen a complaint about it. Or on ext3, which does not support preallocation, and causes gocryptfs to disable it automatically.

Anyway, if you want to benchmark this, pull the btrfs branch and add -noprealloc to line 50 in benchmark.bash: https://github.com/rfjakob/gocryptfs/blob/btrfs/benchmark.bash#L50

In my testing, this gives a massive speedup.

[tyrak]

Awesome! I just tested it, and it makes gocryptfs faster than encfs. I am ready to test it thoroughly now. Thanks!

[rfjakob]

Ok good. I have merged the changes to master and dropped the "btrfs" branch.