cron-apt update logic only allows addition of apt dependencies from /etc/apt/security.list

[redshiftzero]

Description

During QA of 0.7.0, in #3316 we discovered that the securedrop-app-code package was not upgrading. It was held back due to the addition of the libjpeg-dev apt dependency in #3057. Further spelunking by @msheiny indicated that we can only add dependencies in repos listed in /etc/apt/security.list, or existing instances will not update. This means that currently we can't add any apt dependencies in repos not on those lists.

User Stories

As a SecureDrop developer, I want to be able to add new apt dependencies to the securedrop-app-code package so I can ship new features.

[redshiftzero]

My understanding is that the reason to restrict upgrades only to repositories in /etc/apt/security.list was such that critical security upgrades are applied promptly even though this means if a security upgrade causes some other issue, it may break something (a good decision given the nature of the project and the relative risk). Presumably this is also why e.g. unattended-upgrades behaves in the same way in its default configuration - i.e. only doing security updates automatically.

If we permit upgrades for more packages, without doing something like #2106 (proxying all apt packages and pulling the lever to send them to instances e.g. when CI passes) it increases the probability of a production failure, as more untested (untested with SecureDrop) code will be running on instances. Without further investigation, it's unclear to me how much the chance of a failure increases, and thus whether or not this is an acceptable risk. My pause here to ask this question is because both outages in the history of the SecureDrop project have been due to untested updates (#2105, #1364). If we estimate the risk of such failures occurring to be low, then I'd be in favor of permitting upgrades for all installed packages.

Perhaps a lower risk option that still enables us to add apt dependencies in the future, while not opening the door to all updates, might be an idea @msheiny had - to host those packages on FPF's repository. This does introduces some maintenance burden and as such we should only introduce dependencies when we actually need them, as we need to monitor them for updates and update them after CI passes, as we now do with Tor. It is certainly a lesser burden than both proxying all apt updates, and also better than having to recover from an outage.

[msheiny]

If we permit upgrades for more packages, without doing something like #2106 (proxying all apt packages and pulling the lever to send them to instances e.g. when CI passes) it increases the probability of a production failure, as more untested (untested with SecureDrop) code will be running on instances.

I am against this strategy because it would push us further into untested territory -- we would be expanding the number of untested packages that can possibly negatively interact with production deployments in unforeseen ways. I could have sworn that this issue #3135 was also an example of Ubuntu upstream packages breaking things in unforeseen ways (how upstream kernel packages could interact with our kernel story in unforeseen ways) to add to your list. It will also be hard to estimate the risk because frankly we have no idea how future packages could or could not interact with our SD application in weird ways. Is the chance low of this happening, yay probably. The risk though is that we wake up one day and all of our SD servers are down again which I think we want to avoid at all costs. Or more nefariously... an update could possibly break the security model of the machine in a way that isnt readily apparent to an admin/us (like SD servers booting up without grsec enabled).

The second option, hosting the additional apt dependencies for x amount of time - solves this problem immediately and gets us closer to pegging version dependencies. It would be trivial to add update logic into our upcoming new apt deployment process to pull the latest of x dependency and add it to our apt servers.

The only current concern you rightfully bring up .... to monitor for security patches for those new libraries. BUT... with our current setup any security update that pours into upstream ubuntu security should get automatically installed thru cron-apt. As long as we arent pegging our repository as higher priority in the apt dependency list... both package versions will be seen by the system and if something in security.ubuntu.com is higher version.. that should get installed. We should absolutely test/confirm that - but that's potentially a big relief valve for us having to closely monitor and patch for security fixes on those apt deps immediately.

[msheiny]

The only current concern you rightfully bring up

ahaha i just realized this statement reads super condescending . my bad! i didnt mean it like that.

[conorsch]

Largely agreed with the intention to be cautious here, and to avoid pulling updates that we aren't testing. However, it's worth noting that we're already inconsistent on this approach: every time the Ansible logic runs—typically only at install time, but also sporadically thereafter throughout the life of the instance—we perform a safe-upgrade via aptitude, which:

Upgrades installed packages to their most recent version. Installed packages will not be removed unless they are unused (see the section “Managing Automatically Installed Packages” in the aptitude reference manual). Packages which are not currently installed may be installed to resolve dependencies unless the --no-new-installs command-line option is supplied.

In case you were wondering, we do not provide the --no-new-installs flag. So prod instances do not have a predictable baseline of package versions, since significant divergence can occur as a result of the frequency of the install action being run (e.g. to enable new features such as journalist notifications).

So whatever path we take with the upgrade story, we should be consistent. If it's OK to upgrade all packages at install time, then perhaps we should restrict future runs to skip the upgrade logic, and defer entirely to the automatic upgrade logic.

[msheiny]

If it's OK to upgrade all packages at install time,

Just to be clear, I strongly disagree with this practice. We should be locking down and QA'ing the exact dependencies everywhere. So we know what is running in production and all our instances have the same baseline. I consider that situation you just described as a bug and that our ultimate goal should be #3095 so prod instances do not have a predictable baseline of package versions doesnt happen.

I agree we should remain consistent but we are already non-consistent -- so we could either stay on the status quo or start making small steps of improvement 🤷‍♂️

[kushaldas]

Just to be clear, I strongly disagree with this practice. We should be locking down and QA'ing the exact dependencies everywhere. So we know what is running in production and all our instances have the same baseline.

This is one very good use case for Project Atomic (ostree based deployments) :) To know what exactly is running on the servers.