[xenial] File uploads fail on source interface on Xenial, with SecureTemporaryFile-related error

[zenmonkeykstop]

Description

File uploads of any size fail in the Source Interface on a Xenial-based system, with the error message

Server error

Sorry, the website encountered an error and was unable to complete your request.

Turning on logging for the source interface, the following error log is recorded on file submissions:

[Fri Feb 08 00:55:32.641676 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793] [2019-02-08 00:55:32,641] ERROR in app: Exception on /submit [POST]
[Fri Feb 08 00:55:32.641722 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793] Traceback (most recent call last):
[Fri Feb 08 00:55:32.641731 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in wsgi_app
[Fri Feb 08 00:55:32.641738 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     response = self.full_dispatch_request()
[Fri Feb 08 00:55:32.641744 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in full_dispatch_request
[Fri Feb 08 00:55:32.641764 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     rv = self.handle_user_exception(e)
[Fri Feb 08 00:55:32.641771 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1718, in handle_user_exception
[Fri Feb 08 00:55:32.641777 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     reraise(exc_type, exc_value, tb)
[Fri Feb 08 00:55:32.641783 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in full_dispatch_request
[Fri Feb 08 00:55:32.641789 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     rv = self.dispatch_request()
[Fri Feb 08 00:55:32.641795 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in dispatch_request
[Fri Feb 08 00:55:32.641801 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     return self.view_functions[rule.endpoint](**req.view_args)
[Fri Feb 08 00:55:32.641807 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/var/www/securedrop/source_app/decorators.py", line 12, in decorated_function
[Fri Feb 08 00:55:32.641813 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     return f(*args, **kwargs)
[Fri Feb 08 00:55:32.641819 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/var/www/securedrop/source_app/main.py", line 158, in submit
[Fri Feb 08 00:55:32.641825 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     fh.stream))
[Fri Feb 08 00:55:32.641831 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/var/www/securedrop/store.py", line 139, in save_file_submission
[Fri Feb 08 00:55:32.641837 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     with SecureTemporaryFile("/tmp") as stf:  # nosec
[Fri Feb 08 00:55:32.641843 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/var/www/securedrop/secure_tempfile.py", line 49, in __init__
[Fri Feb 08 00:55:32.641849 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     self.create_key()
[Fri Feb 08 00:55:32.641855 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/var/www/securedrop/secure_tempfile.py", line 66, in create_key
[Fri Feb 08 00:55:32.641861 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     self.initialize_cipher()
[Fri Feb 08 00:55:32.641867 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/var/www/securedrop/secure_tempfile.py", line 72, in initialize_cipher
[Fri Feb 08 00:55:32.641873 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     self.cipher = Cipher(AES(self.key), CTR(self.iv), default_backend())
[Fri Feb 08 00:55:32.641880 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
[Fri Feb 08 00:55:32.641886 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     from cryptography.hazmat.backends.openssl.backend import backend
[Fri Feb 08 00:55:32.641892 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
[Fri Feb 08 00:55:32.641899 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     from cryptography.hazmat.backends.openssl.backend import backend
[Fri Feb 08 00:55:32.641905 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 23, in <module>
[Fri Feb 08 00:55:32.641911 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793]     from cryptography.hazmat.backends.openssl import aead
[Fri Feb 08 00:55:32.641931 2019] [wsgi:error] [pid 2798:tid 3926105958144] [remote 127.0.0.1:10793] ImportError: cannot import name aead
[Fri Feb 08 00:55:32.645154 2019] [wsgi:error] [pid 2798:tid 3926105958144] Exception KeyError: KeyError('file',) in <bound method SecureTemporaryFile.__del__ of <secure_tempfile.SecureTemporaryFile object at 0x3920ae41890>> ignored

Steps to Reproduce

Environment: 5-Series NUCs running Ubuntu 16.04.5 as base OS (note that system was installed using playbook described at bottom of #4109 to get around missing package issues)

• Open the Source Interface and browse to submissions page
• Submit a file

Expected Behavior

If file is <500MB, submission completes without error

Actual Behavior

Regardless of size, file submission fails as described above.

Comments

I also saw a slightly different traceback a couple of times, failing with an openssl.so memory protection error, but I haven't been able to reproduce it since.

It's entirely possible that this is a result of the installation procedure for this test instance, see #4109 for details. But it doesn't look like there are missing packages that could account for the error.

[kushaldas]

I can verify the same error.

[kushaldas]

Looks like grsec in play:

[Fri Feb 08 11:49:03.707586 2019] [wsgi:error] [pid 6610:tid 3887623821056]     from cryptography.hazmat.bindings._openssl import ffi, lib
[Fri Feb 08 11:49:03.707590 2019] [wsgi:error] [pid 6610:tid 3887623821056] ImportError: /usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_openssl.so: cannot change memory protections: Permission denied

[kushaldas]

The following commands fixed it for me:

# service apache2 stop
# paxctl -cm /usr/sbin/apache2
file /usr/sbin/apache2 had a PT_GNU_STACK program header, converted
# service apache2 start

@zenmonkeykstop @conorsch Not sure why are we missing this?

[conorsch]

Able to reproduce. Tested with a 2MB file and 50MB file, both failed reliably. The error is not triggered when submitting text-only, one must upload a file in order to trigger the behavior.

As far as resolution goes, @kushaldas's suggestion does work, but will not protect us against out of band updates to the apache2 binary. Since we have automatic nightly updates to apply security fixes, apache is likely to be updated out of sync with the securedrop-app-code package, and during such an upgrade, the custom PaX flags we set would be wiped when the binary is replaced, breaking the service again.

Let's instead aim to use paxctld, which is available in the Ubuntu Xenial apt repos, as that service aims to solve the problem we're encountering. Thanks to #4080, we can insert the dependency only on Xenial, and chain the config munging on whether the package is present.

Not sure why are we missing this?

A great question! We've never had to manage PaX flags on the apache binary under Trusty, but we do routinely manage flags on Python binaries for applications we maintain on non-Trusty platforms. At a glance, I suspect the divergence in openssl versions between Trusty & Xenial would explain the differing behavior, but that's only a hypothesis at this point.

Will submit a PR with the paxctld proposal described above.