Template consolidation via GUI updater#619
Conversation
| {% endif %} | ||
|
|
||
| {% if grains['id'] == "sd-log-buster-template" %} | ||
| {% if grains['id'] in ["sd-small-buster-template", "sd-large-buster-template"] %} |
There was a problem hiding this comment.
Do we need to install redis in large template? Should only be required for receiver of logs.
| - sd-logging-setup | ||
| - sd-workstation-template-files | ||
| - sd-devices-files | ||
| - sd-viewer-files |
There was a problem hiding this comment.
TODO: add mime handling to small & large templates, also to sd-gpg appvm
|
Test plan above working well, seeing expected result after updates. Couple of issues though:
|
conorsch
force-pushed
the
608-import-export-security-rebased
branch
from
c69db7e to
6768811
Compare
|
Rebased to stay current with "main". To proceed with the securedrop-export changes, we'll need to review and merge freedomofpress/securedrop-apt-test#67, then use the template-consolidation channel to evaluate provisioning, upgrade, and config test coverage here. After that, I plan to document the |
|
On latest revisions, still seeing two failing tests: Those are surely related to the recent export changes in freedomofpress/securedrop-export#65, so let's investigate. |
…pdate logic doesn't break)
The sd-proxy mimetype handlers will be configured via the `securedrop-workstation-config` package, same as others.
conorsch
force-pushed
the
608-import-export-security-rebased
branch
from
e3ae0ae to
3671717
Compare
conorsch
force-pushed
the
608-import-export-security-rebased
branch
from
3671717 to
63b42bf
Compare
There was a problem hiding this comment.
So far, testing has been successful in both the new install case and the upgrade case (in the dev environment):
- upgrade testing from
mainbased on the test plan, with all tests passing - provisioning from scratch on this branch, with all tests passing
I did notice however a minor deviation /usr/share/applications/mimeapps.list is present on the small template but not the large template. this is because 1. freedomofpress/securedrop-client#1160 is not yet merged and 2. https://github.com/freedomofpress/securedrop-dev-packages-lfs/tree/main/workstation/template-consolidation does not create a version of the package greater than 2.1 with these changes (ie removing mimeapps.list in the template in which securedrop-client is installed).
The presence of this file is helpful to ensure defense-in-depth against mime handling misconfiguration (defaulting to a safe, open-in-dvm option), but it may be best handled in securedrop-workstation-config package (or the salt logic) to ensure converge accross both templates
zenmonkeykstop
force-pushed
the
608-import-export-security-rebased
branch
2 times, most recently
from
bd2f888 to
ff2a599
Compare
We'll be posting the rc1 package to yum-test to aid in QA. Not using a tag, as described in docs [0], since we're postponing merge of feature branch [1] until QA is finished, given the scope of changes. [0] https://github.com/freedomofpress/securedrop-workstation/wiki/Building-securedrop-workstation-dom0-config-RPM-package [1] #619
Based on changes presented in freedomofpress/securedrop-workstation#619 Note that no tag is used here, since QA is happening against the feature branch. In lieu of a tag, I verified a commit in the build logs, but that commit hash will be lost with subsequent rebases. Still, the sig verification is useful for review of the package.
For QA during review of [0]. See build logs at [0] freedomofpress/securedrop-workstation#619 [1] freedomofpress/build-logs@ef65668
|
Ran through the test plan for "Happy path" in https://github.com/freedomofpress/securedrop-workstation/wiki/0.5.0-Test-Plan. Everything worked smoothly. Included some minimal acceptance testing incluing login, reply, download, decrypt, and open-in-dvm. Full results:
|
conorsch
changed the title
|
Ran into some issues when running I propose we:
EDIT: (In the original comment, I had not pulled latest changes) After pulling latest changes, i can confirm |
|
Great, thanks for confirming @emkll! In standup today we agreed it's time to starting merging. @zenmonkeykstop is going to move the qa-switch material out of the RPM, then I'll drop the temporary commits—both here and in the related PRs—and promote to final "ready-for-review" status, so we can get approvals in. |
With this change, all SDW VMs are using the new consolidated small/large
templates.
Reuses salt state files in top file
The new small & large templates don't need to be configured in their own
state files, we can just add references to the top file to assign the
old states per vm and consolidate them.
Removes unconsolidated templates from config
Don't bother to create them if we're not planning on using them.
Updates make-dev target to use new consolidated templates
sd-gpg: wait for consolidated templates
The consolidated templates spike was handled in a different SLS file,
which sd-gpg wasn't depending on it. That makes for a race, so let's
update the dependency list to ensure proper ordering.
Removed old templates and updated tests.
Removed definitions for sd-{app,log,proxy}-buster-template VMs.
Added back dom0/mimeapps.list, as it's used by config tests.
Updated tests to refer to new templates.
Updated logging setup to ensure correct config present on non sd-log VMs
Consolidates templates via handle-upgrade script
The "securedrop-handle-upgrade" script was added in order to migrate
from Stretch-based to Buster-based templates. With some small changes to
the expected template pattern, we can leverage the same script to handle
consolidating the templates.
This only handles *creation* of the VMs, it doesn't sort out the apt
packages and config logic specific to each Template/App combination.
Additional logic will need to be added to the updater to support
hands-off consolidation.
Uses new pkg securedrop-workstation-viewer
Supersedes the securedrop-workstation-sd-svs-disp package, to make the
transition to consolidated templates a bit easier, by avoiding dpkg
conflicts altogether.
Tests for securedrop-workstation-svs-disp absence
The package "securedrop-workstation-svs-disp" has been superseded by
"securedrop-workstation-viewer", so in testing for presence of one,
ensure the absence of the other.
Updates the config tests to permit testing package absence; before it
was showing a confusing stack trace if package was missing (so error
rather than failure).
Forces template setting on sd-log
Using "prefs" rather than "present" so that the value is updated, not
merely set at VM creation time.
Sets default mimetype handler for sd-proxy
Uses the new sd-mime-handling state and applies it to sd-proxy, so that
symlinks for mimetypes are handled appropriately post-template-
consolidation. Requires packaging changes that are for now only in a
feature branch on the dev packages repo, served via apt-test.
Includes a small fix to ensure that the files in private volume are
owned by normal user, not root.
Removes unused sd-proxy files
The do-not-open here files are left over from a prototype of the
workstation, no longer required.
Tests sd-proxy: use open-in-dvm for mimetypes
We've ditched the do-not-open-here logic, so sd-proxy should now default
to using open-in-dvm for all filetypes. The config tests now reflect
this. Copy/pasted from the sd-app tests, didn't bother to refactor to
make it DRY.
Fix mimetype private volume perms
Using "mode" and "makedirs" together for a symlink led to a broken
config: Salt was creating the parent directories with mode 644, so they
weren't traversable, so the mimeapps.list file couldn't be read by
normal user. Fixed.
Handle upgrade script: remove old templates
Leverages the "remove" subcommand for the securedrop-handle-upgrade
script to purge the old, non-consolidated TemplateVMs after migrating.
Moves sd-proxy config to private volume
Requires package update to securedrop-proxy, so that the RPC config at
/etc/qubes-rpc/securedrop.Proxy references the new filepath.
Includes a new test for that file, since we weren't explicitly examining
its contents before.
Removes legacy sd-proxy config file
It shouldn't ever be present post-consolidated, given that the newly
created VM never had the /etc/ path configured, but adding anyway as a
defensive measure, and also to get the tests passing sooner.
Moves consolidated templates into single state
We originally had the consolidated templates in their own salt state
file, to aid in development, so we could target just that one component.
Folding into the pre-existing state file for the base template now, and
cleaning up the extra references, mostly to minimize the diff and aid in
review.
Follow up to [PR]. When running "make clone && make prep-dev" repeatedly in dom0 during development, the dnf cache in dom0 gets out of sync, and gets confused about whether the "securedrop-workstation-dom0-config" package is installed. Running "sudo dnf clean all" purges the cache and keeps things working smoothly.
Adds a "securedrop-check-migration" script that will drop a flag in /tmp/, for the updater's consumption, if the updater should rerun all salt states, rather than just package upgrades. Required for significant changes like template consolidation.
As explained in the comment, we accept the risk of a hardcoded tempfile in dom0, because it's a single-user system, isolated from the other VM components. At a later date, we'll likely port the check-migration logic to pure python, but unfortunately 'import qubesadmin' would break the mocked CI tests for the launcher and updater.
Follow-up to [0]. Post-consolidation, we can expect the /etc/profile.d/ path to be present on all systems, but only on sd-app should it return "sd-gpg" rather than an empty string. [0] #623
Ensuring that we're able to ship unattended migrations in the future. It's important that *first* we update the dom0 RPM, if required, then re-apply the dom0 state. After that, we'll check whether a migration is requested (that's the clincher), then either run a full-state apply or upgrade packages within each TemplateVM, depending on the check. Tinkered with the progress bars accordingly. It's not ideal, since the full-state run blocks the progress bar.
Squashes a few related commits for the qa-switcher logic, including the final change prior to merge that moved the files to a separate location so they wouldn't be included in the rc RPMs.
We'll be posting the rc1 package to yum-test to aid in QA. Not using a tag, as described in docs [0], since we're postponing merge of feature branch [1] until QA is finished, given the scope of changes. [0] https://github.com/freedomofpress/securedrop-workstation/wiki/Building-securedrop-workstation-dom0-config-RPM-package [1] #619
Including a "0.5.0" message as a start, even though we're only on rc1.
conorsch
force-pushed
the
608-import-export-security-rebased
branch
from
3d16c1e to
fc35345
Compare
|
Removed temporary commits for final review. Preserved the Ready for final review! |
emkll
left a comment
emkll
left a comment
There was a problem hiding this comment.
diff from the previous review looks good to me https://github.com/freedomofpress/securedrop-workstation/compare/3d16c1e..fc35345
With the associated packages changes merged in other repos, this should be good to merge
|
All nightlies/packages and changes required for this branch have been updated on apt-test https://apt-test.freedom.press/pool/main/ |
As part of review, we've moved candidate packages from the "template-consolidation" channel on apt-test to "main" on apt-test. Therefore the qa-switch tool must be updated accordingly.
emkll
left a comment
emkll
left a comment
There was a problem hiding this comment.
Changes to qa-switcher look good to me
Now that we've merged the template-consolidation PR [0], we can resume the normal rc build procedure documented in the wiki [1]. [0] #619 [1] https://github.com/freedomofpress/securedrop-workstation/wiki/Building-securedrop-workstation-dom0-config-RPM-package
We'll be posting the rc1 package to yum-test to aid in QA. Not using a tag, as described in docs [0], since we're postponing merge of feature branch [1] until QA is finished, given the scope of changes. [0] https://github.com/freedomofpress/securedrop-workstation/wiki/Building-securedrop-workstation-dom0-config-RPM-package [1] freedomofpress/securedrop-workstation#619
Now that we've merged the template-consolidation PR [0], we can resume the normal rc build procedure documented in the wiki [1]. [0] freedomofpress/securedrop-workstation#619 [1] https://github.com/freedomofpress/securedrop-workstation/wiki/Building-securedrop-workstation-dom0-config-RPM-package
…y-rebased Template consolidation via GUI updater
3 participants
Status
Ready for review
Description of Changes
Closes #471.
sd-small-buster-templateandsd-large-buster-template.Depends on multiple closely related PRs:
Template consolidation test plan
The procedure for upgrading prod machines will be as follows:
sudo qubes-dom0-update -y. Doing so will obtain the latest GUI updater code.Given the special case of requesting user action, we'll need to test both the "happy" path, where users follow our advice exactly, and the "sad" path, where special action is not taken.
See detailed test plan for the upcoming 0.5.0 release: https://github.com/freedomofpress/securedrop-workstation/wiki/0.5.0-Test-Plan Complete only the sections under Scenario: Template consolidation. The other scenarios we'll handle as part of QA, after merging this and related PRs.
QA
Writing a separate "QA" checklist because merging this and its related PR requires careful orchestration. Prior to merge of this PR, there are test-only commits marked "TEMPORARY" that should be removed. Doing so is required to use the "main" channel of the apt-test repository. Right now, packages are only present in the "template-consolidation" channel to facilitate testing.
The following PRs must be reviewed, then merged together:
Checklist
If you have made code changes
make flake8) passes in the development environment (this box maybe left unchecked, as
flake8also runs in CI)If you have made changes to the provisioning logic
All tests (
make test) pass indom0of a Qubes installThis PR adds/removes files, and includes required updates to the packaging
logic in
MANIFEST.inandrpm-build/SPECS/securedrop-workstation-dom0-config.specSwitching a prod install to update from QA repositories
For testing purposes it may be necessary to modify a prod installation to use QA repos. The process of doing so is threefold:
apt-test.freedom.pressand add the testing key in all Debian-based SD templatesyum.securedrop.orgtoyum-test.securedrop.orgTo do so safely using Salt, as root:
/home/user/securedrop-workstation/scripts/qa-switch.tar.gzin /srv/saltbash /srv/salt/consolidation-qa-switch.sh