[0.5.0] Several mimetype-related tests are failing

[conorsch]

Last tested on PR #632 (which will likely become 0.5.0-rc3).

failing tests output

======================================================================
ERROR: test_open_in_dvm_desktop (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_sd_devices.py", line 40, in test_open_in_dvm_desktop
    contents = self._get_file_contents("/usr/share/applications/open-in-dvm.desktop")
  File "/home/user/securedrop-workstation/tests/base.py", line 62, in _get_file_contents
    contents = subprocess.check_output(cmd).decode("utf-8")
  File "/usr/lib64/python3.5/subprocess.py", line 316, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['qvm-run', '-p', 'sd-devices-dvm', 'sudo /bin/cat /usr/share/applications/open-in-dvm.desktop']' returned non-zero exit status 1

======================================================================
FAIL: test_mime_types (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_sd_devices.py", line 37, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: '' != 'open-in-dvm.desktop'
+ open-in-dvm.desktop

======================================================================
FAIL: test_mime_types (test_viewer.SD_Viewer_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_viewer.py", line 45, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: 'libreoffice-calc.desktop' != 'libreoffice-base.desktop'
- libreoffice-calc.desktop
?             ^ ^^
+ libreoffice-base.desktop
?             ^ ^^


======================================================================
FAIL: test_all_sd_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_vms_platform.py", line 169, in test_all_sd_vms_uptodate
    self._ensure_packages_up_to_date(vm)
  File "/home/user/securedrop-workstation/tests/test_vms_platform.py", line 95, in _ensure_packages_up_to_date
    self.assertEqual(results, "Listing...", fail_msg)
AssertionError: 'Listing...\nsecuredrop-workstation-config/unknown 0[43 chars]ter]' != 'Listing...'
- Listing...
?           -
+ Listing...- securedrop-workstation-config/unknown 0.1.5+buster all [upgradable from: 0.1.3+buster] : Unapplied updates for VM 'sd-log'

----------------------------------------------------------------------
Ran 79 tests in 354.716s

FAILED (failures=3, errors=1)
Makefile:109: recipe for target 'test' failed
make: *** [test] Error 1

Let's make sure to get these down to zero prior to releasing 0.5.0 final.

[zenmonkeykstop]

Output on a fresh make dev for #632 for me:

Details

[kog@dom0 securedrop-workstation]$ make test
python3 -m unittest discover -v tests
test_gpg_domain_configured (test_app.SD_App_Tests) ... ok
test_logging_configured (test_app.SD_App_Tests) ... ok
test_mimeapps (test_app.SD_App_Tests) ... ok
test_mimeapps_functional (test_app.SD_App_Tests) ... ok
test_open_in_dvm_desktop (test_app.SD_App_Tests) ... ok
test_sd_client_apparmor (test_app.SD_App_Tests) ... ok
test_sd_client_config (test_app.SD_App_Tests) ... ok
test_sd_client_dependencies_installed (test_app.SD_App_Tests) ... ok
test_sd_client_package_installed (test_app.SD_App_Tests) ... ok
test_Templates_cleaned_up (test_dom0_config.SD_Qubes_Dom0_Templates_Tests) ... ok
test_vms_to_update_are_tagged (test_dom0_config.SD_Qubes_Dom0_Templates_Tests) ... ok
test_rpm_repo_config (test_dom0_rpm_repo.SD_Dom0_Rpm_Repo_Tests) ... ok
test_rpm_repo_public_key (test_dom0_rpm_repo.SD_Dom0_Rpm_Repo_Tests) ... ok
test_gpg_domain_configured (test_gpg.SD_GPG_Tests) ... ok
test_logging_disabled (test_gpg.SD_GPG_Tests) ... ok
test_sd_gpg_timeout (test_gpg.SD_GPG_Tests) ... ok
test_we_have_the_key (test_gpg.SD_GPG_Tests) ... ok
test_gpg_domain_configured (test_log_vm.SD_Log_Tests) ... ok
test_log_dirs_properly_named (test_log_vm.SD_Log_Tests) ... ok
test_log_utility_installed (test_log_vm.SD_Log_Tests) ... ok
test_logs_are_flowing (test_log_vm.SD_Log_Tests) ... ok
test_redis_service_running (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_has_no_custom_rsyslog (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_package_installed (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_redis_is_installed (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_service_running (test_log_vm.SD_Log_Tests) ... ok
test_do_not_open_here (test_proxy_vm.SD_Proxy_Tests) ... ok
test_gpg_domain_configured (test_proxy_vm.SD_Proxy_Tests) ... ok
test_logging_configured (test_proxy_vm.SD_Proxy_Tests) ... ok
test_mime_types (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_package_installed (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_rpc_spec (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_writable_config_dir (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_yaml_config (test_proxy_vm.SD_Proxy_Tests) ... ok
test_whonix_ws_repo_absent (test_proxy_vm.SD_Proxy_Tests) ... ok
test_Policies (test_qubes_rpc.SD_Qubes_Rpc_Tests) ... ok
test_current_fedora_for_sys_vms (test_qubes_vms.SD_Qubes_VM_Tests) ... ok
test_current_whonix_vms (test_qubes_vms.SD_Qubes_VM_Tests) ... ok
test_files_are_properly_copied (test_sd_devices.SD_Devices_Tests) ... ok
test_gpg_domain_configured (test_sd_devices.SD_Devices_Tests) ... ok
test_logging_configured (test_sd_devices.SD_Devices_Tests) ... ok
test_mime_types (test_sd_devices.SD_Devices_Tests) ... FAIL
test_open_in_dvm_desktop (test_sd_devices.SD_Devices_Tests) ... /bin/cat: /usr/share/applications/open-in-dvm.desktop: No such file or directory
ERROR
test_sd_export_package_installed (test_sd_devices.SD_Devices_Tests) ... ok
test_accept_sd_xfer_extracted_file (test_sd_whonix.SD_Whonix_Tests) ... ok
test_gpg_domain_configured (test_sd_whonix.SD_Whonix_Tests) ... ok
test_logging_configured (test_sd_whonix.SD_Whonix_Tests) ... ok
test_sd_whonix_repo_enabled (test_sd_whonix.SD_Whonix_Tests) ... ok
test_sd_whonix_verify_tor_config (test_sd_whonix.SD_Whonix_Tests) ... ok
test_v3_auth_private_file (test_sd_whonix.SD_Whonix_Tests) ... skipped 'Onion v3 address is not setup'
test_whonix_torrc (test_sd_whonix.SD_Whonix_Tests) ... ok
test_rpm_repo_public_key (test_sys_firewall.SD_Sys_Firewall_Tests) ... ok
test_rpm_repo_public_key_script (test_sys_firewall.SD_Sys_Firewall_Tests) ... ok
test_files_are_properly_copied (test_sys_usb.SD_SysUSB_Tests) ... ok
test_gpg_domain_configured (test_viewer.SD_Viewer_Tests) ... ok
test_logging_configured (test_viewer.SD_Viewer_Tests) ... ok
test_mime_types (test_viewer.SD_Viewer_Tests) ... FAIL
test_redis_packages_not_installed (test_viewer.SD_Viewer_Tests) ... ok
test_sd_viewer_evince_installed (test_viewer.SD_Viewer_Tests) ... ok
test_sd_viewer_libreoffice_installed (test_viewer.SD_Viewer_Tests) ... ok
test_sd_viewer_metapackage_installed (test_viewer.SD_Viewer_Tests) ... ok
test_expected (test_vms_exist.SD_VM_Tests) ... ok
test_sd_app_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_gpg_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_log_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_proxy_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_proxy_template (test_vms_exist.SD_VM_Tests) ... ok
test_sd_viewer_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_whonix_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_workstation_template (test_vms_exist.SD_VM_Tests) ... ok
test_all_fedora_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_jessie_backports_disabled (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_sd_vm_apt_sources (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_sd_vm_platforms (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_sd_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests) ... FAIL
test_debian_keyring_config (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_dispvm_default_platform (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_sd_proxy_template (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_sys_vms_use_supported_fedora (test_vms_platform.SD_VM_Platform_Tests) ... ok

======================================================================
ERROR: test_open_in_dvm_desktop (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_sd_devices.py", line 40, in test_open_in_dvm_desktop
    contents = self._get_file_contents("/usr/share/applications/open-in-dvm.desktop")
  File "/home/kog/securedrop-workstation/tests/base.py", line 62, in _get_file_contents
    contents = subprocess.check_output(cmd).decode("utf-8")
  File "/usr/lib64/python3.5/subprocess.py", line 316, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['qvm-run', '-p', 'sd-devices-dvm', 'sudo /bin/cat /usr/share/applications/open-in-dvm.desktop']' returned non-zero exit status 1

======================================================================
FAIL: test_mime_types (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_sd_devices.py", line 37, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: '' != 'open-in-dvm.desktop'
+ open-in-dvm.desktop

======================================================================
FAIL: test_mime_types (test_viewer.SD_Viewer_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_viewer.py", line 45, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: 'libreoffice-calc.desktop' != 'libreoffice-base.desktop'
- libreoffice-calc.desktop
?             ^ ^^
+ libreoffice-base.desktop
?             ^ ^^


======================================================================
FAIL: test_all_sd_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_vms_platform.py", line 169, in test_all_sd_vms_uptodate
    self._ensure_packages_up_to_date(vm)
  File "/home/kog/securedrop-workstation/tests/test_vms_platform.py", line 95, in _ensure_packages_up_to_date
    self.assertEqual(results, "Listing...", fail_msg)
AssertionError: 'Listing...\nsecuredrop-workstation-config/unknown 0[43 chars]ter]' != 'Listing...'
- Listing...
?           -
+ Listing...- securedrop-workstation-config/unknown 0.1.5+buster all [upgradable from: 0.1.3+buster] : Unapplied updates for VM 'sd-log'

----------------------------------------------------------------------
Ran 79 tests in 162.291s

• The error and first failure related to open-in-dvm.desktop are probably a result of consolidation changes - investigation is required to see if the tests need changing or if the config isn't as expected.
• The failure related to libreofffice-base.deskstop looks like a result of bad test data and updates to libreoffice itself, those mime types are now associated with the specific application desktop file that handles them as opposed to the base one.
• The last failure looks to be a result of using nightlies and is probably safe to ignore

[emkll]

I have also observed these errors locally, however after running the updater (unclear and unlikely that had an impact) and shutting down all the workstation VMs, most tests were once again passing on the branch in 632.

It seems like the issue is the DispVM templates which weren't shutdown after the initial provisioning run: sd-devices-dvm and sd-viewer were out of sync with the template (they were using config 1.3 which is the one "baked into" the base template, which means securedrop-workstation-buster was not shutdown after updating and before it is clone to the small and large templates.

We should check the shutdown order within the GUI updater, and since a normal user would have to run through an updater cycle to get their workstation in order after running the manual --apply command, this could address the situation in non-dev scenarios, but would obvious be an issue for dev

[conorsch]

Thanks @emkll, that's helpful. A full host reboot likely would have straightened me out, I'll confirm that today.

[conorsch]

Encountered again on 0.5.0-rc3 as part of QA for fresh install. Ran the updater, then manually restarted the VMs, and all tests were passing. Running make-dev and make-test from the same code, via the main branch, showed all tests passing immediately, so I suspect the state drift is related to our use of the qa-switch.sh util mandated by the test plan. Removing the "release-blocker" label for 0.5.0, but leaving open, so we can observe behavior of the prod packages and decide on whether a 0.5.1 is warranted.

[emkll]

Related to freedomofpress/securedrop-updater#35

[emkll]

Closing, it seems like freedomofpress/securedrop-updater#35 is the underlying issue here, and 0.5.0 has been released.

[conorsch]

FWIW I'm no longer able to reproduce these test failures. On latest main. i.e. 6b4896d, I'm seeing 100% of tests passing. freedomofpress/securedrop-updater#35 should still be addressed, but at least our tests are passing again.