Panic in xfstests generic/075

[rfjakob]

Backtrace:

$ sudo ./check-gocryptfs generic/075
[...]
2016/10/26 21:48:17 Hit an all-zero nonce in block 987. This MUST NOT happen!
panic: Hit an all-zero nonce in block 987. This MUST NOT happen!

goroutine 8 [running]:
panic(0x5e1220, 0xc42020c240)
    /usr/local/go/src/runtime/panic.go:500 +0x1a1
log.Panicf(0x638859, 0x38, 0xc42021f7d8, 0x1, 0x1)
    /usr/local/go/src/log/log.go:327 +0xe3
github.com/rfjakob/gocryptfs/internal/contentenc.(*ContentEnc).DecryptBlock(0xc420088410, 0xc4204b7360, 0x48e, 0x50a0, 0x3db, 0xc420239020, 0x10, 0x10, 0xc4201d8000, 0x1000, ...)
    /home/jakob/src/github.com/rfjakob/gocryptfs/internal/contentenc/content.go:119 +0x7a0
github.com/rfjakob/gocryptfs/internal/contentenc.(*ContentEnc).DecryptBlocks(0xc420088410, 0xc42049c000, 0x1b7ee, 0x20400, 0x3db, 0xc420239020, 0x10, 0x10, 0x0, 0x0, ...)
    /home/jakob/src/github.com/rfjakob/gocryptfs/internal/contentenc/content.go:83 +0x1ae
github.com/rfjakob/gocryptfs/internal/fusefrontend.(*file).doRead(0xc4200603c0, 0x3c0000, 0x20000, 0xc42021fd18, 0x3, 0x3, 0x18)
    /home/jakob/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file.go:171 +0x946
github.com/rfjakob/gocryptfs/internal/fusefrontend.(*file).Read(0xc4200603c0, 0xc42047c000, 0x20000, 0x20000, 0x3c0000, 0x0, 0x0, 0xc400000000)
    /home/jakob/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file.go:207 +0x31a
github.com/hanwen/go-fuse/fuse/pathfs.(*pathInode).Read(0xc420016ea0, 0x926bc0, 0xc4200603c0, 0xc42047c000, 0x20000, 0x20000, 0x3c0000, 0xc4201f63e0, 0xc400020000, 0xc42047c000, ...)
    /home/jakob/src/github.com/hanwen/go-fuse/fuse/pathfs/pathfs.go:731 +0x65
github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Read(0xc420017740, 0xc4201f63c8, 0xc42047c000, 0x20000, 0x20000, 0x20000, 0x4ed8e5, 0x0)
    /home/jakob/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:455 +0xda
github.com/hanwen/go-fuse/fuse.doRead(0xc42008a540, 0xc4201f6240)
    /home/jakob/src/github.com/hanwen/go-fuse/fuse/opcode.go:315 +0x9d
github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc42008a540, 0xc4201f6240, 0xc4201f6240)
    /home/jakob/src/github.com/hanwen/go-fuse/fuse/server.go:398 +0xd3
github.com/hanwen/go-fuse/fuse.(*Server).loop(0xc42008a540, 0x5de801)
    /home/jakob/src/github.com/hanwen/go-fuse/fuse/server.go:377 +0x175
created by github.com/hanwen/go-fuse/fuse.(*Server).readRequest
    /home/jakob/src/github.com/hanwen/go-fuse/fuse/server.go:285 +0x367

Seems to be a bug in tmpfs, reported to linux-kernel at http://www.spinics.net/lists/kernel/msg2370127.html

[rfjakob]

Workaround commited as 012152f .

[rfjakob]

Released in gocryptfs v1.1.1.

[rfjakob]

Hugh Dickins, the tmpfs maintainer, responded to the email and confirmed the issue:
https://lore.kernel.org/all/18e9fa0f-ec31-9107-459c-ae1694503f87@gmail.com/t/ , mirror:
http://www.spinics.net/lists/kernel/msg2374289.html

Copied below for archival.

Re: tmpfs returns incorrect data on concurrent pread() and truncate()

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

To: Jakob Unterwurzacher jakobunt@xxxxxxxxx
Subject: Re: tmpfs returns incorrect data on concurrent pread() and truncate()
From: Hugh Dickins hughd@xxxxxxxxxx
Date: Tue, 1 Nov 2016 16:51:30 -0700 (PDT)
Cc: linux-kernel@xxxxxxxxxxxxxxx, linux-fsdevel@xxxxxxxxxxxxxxx
User-agent: Alpine 2.11 (LSU 23 2013-08-11)
On Wed, 26 Oct 2016, Jakob Unterwurzacher wrote:

tmpfs seems to be incorrectly returning 0-bytes when reading from
a file that is concurrently being truncated.

That is an interesting observation, and you got me worried;
but in fact, it is not a tmpfs problem: if we call it a
problem at all, it's a VFS problem or a userspace problem.

You chose a ratio of 3 preads to 1 ftruncate in your program below:
let's call that the Unterwurzacher Ratio, 3 for tmpfs; YMMV, but for
me 4 worked well to show the same issue on ramfs, and 15 on ext4.

The Linux VFS does not serialize reads against writes or truncation
very strictly: it's unusual to need that serialization, and most
users prefer maximum speed to the additional locking, or intermediate
buffering, that would be required to avoid the issue you've seen.

This is causing crashes in gocryptfs, a cryptographic FUSE overlay,
when it reads a nonce from disk that should absolutely positively
never be all-zero.

I don't think that there will be much appetite for changing the
kernel's VFS to prevent that. I hope that gocryptfs can provide
the serialization that it needs for itself, or otherwise handle
those zeroes without crashing.

(Though if something is free to truncate at an awkward moment,
then I imagine it can also mess things up by writing wrong data.)

I have written a reproducer in C that triggers this issue on
both boxes I tested, Linux 4.7.2 and 3.16.7, both amd64.

It can be downloaded from here:

https://gist.github.com/rfjakob/d01281c737db38075767f90bf03fc475

or, alternatively, I have attached it to this email at the bottom.

The reproducer:

1. Creates a 10MB file filled with 'x' at /dev/shm/x
2. Spawns a thread that truncates the file 3 bytes at a time
3. Spawns another thread that pread()s the file 1 byte at a time
   starting from the top
4. Prints "wrong data" whenever the pread() gets something that
   is not 'x' or an empty result.

Nice work, thank you, this helped me a lot.

Hugh