Fixture in functional tests fails randomly in focal-app-tests

[DrGFreeman]

Description

The missing_msg_file fixture used to create a submission and delete the associated file in the store for tests in tests/functional/test_journalist.py::TestJournalistMissingFile fails in a non-deterministic manner in the focal-app-test CI job.

Steps to Reproduce

Run focal-app-test CI job and cross your fingers.

Expected Behavior

Fixture does not fail.

Actual Behavior

Fixture fails when asserting there is only one file in the store (expected condition). Different failures tracebacks indicate different number of files in the store:

==================================== ERRORS ====================================
___ ERROR at setup of TestJournalistMissingFile.test_download_source_unread ____

self = <tests.functional.test_journalist.TestJournalistMissingFile object at 0x7f80ea826cd0>

    @pytest.fixture(scope="function")
    def missing_msg_file(self):
        """Fixture to setup the message with missing file used in the following tests."""
        # Submit a message
        self._source_visits_source_homepage()
        self._source_chooses_to_submit_documents()
        self._source_continues_to_submit_page()
        self._source_submits_a_message()
        self._source_logs_out()
    
        # Remove the message file from the store
        storage_path = Path(self.journalist_app.storage.storage_path)
        msg_files = [p for p in storage_path.rglob("*") if p.is_file()]
>       assert len(msg_files) == 1
E       assert 2 == 1
E         +2
E         -1

/home/circleci/project/securedrop/tests/functional/test_journalist.py:105: AssertionError
---------------------------- Captured stdout setup -----------------------------
 * Serving Flask app "source_app" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Serving Flask app "journalist_app" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
_ ERROR at setup of TestJournalistMissingFile.test_select_source_and_download_all _

self = <tests.functional.test_journalist.TestJournalistMissingFile object at 0x7f80e9930be0>

    @pytest.fixture(scope="function")
    def missing_msg_file(self):
        """Fixture to setup the message with missing file used in the following tests."""
        # Submit a message
        self._source_visits_source_homepage()
        self._source_chooses_to_submit_documents()
        self._source_continues_to_submit_page()
        self._source_submits_a_message()
        self._source_logs_out()
    
        # Remove the message file from the store
        storage_path = Path(self.journalist_app.storage.storage_path)
        msg_files = [p for p in storage_path.rglob("*") if p.is_file()]
>       assert len(msg_files) == 1
E       assert 3 == 1
E         +3
E         -1

/home/circleci/project/securedrop/tests/functional/test_journalist.py:105: AssertionError

Comments

Observations so far:

1. No failures observed on Xenial (app-test CI job).
2. Cannot reproduce in Focal dev container.
3. So far failures occurred either on the first setup of the fixture (i.e. first test in TestJournalistMissingFile) or on the first and second setups of the fixture.

One hypothesis for the root cause of the failures is that the tests in CI run in parallel and the functional tests use a fixed directory for the store (/tmp/securedrop/store) instead of using the config fixture defined in conftest.py which uses tmpdir as base directory for the store. If multiple tests run in parallel, it is therefore possible that files from multiple tests are stored simultaneously in the store directory.

The above hypothesis does not however explain why the tests would only fail on Focal and why they always seem to fail on the first or first and second tests out of the five that use the fixture...

Potential solutions

1. Use unique storage directory for each test like in the non-functional tests (major refactoring required and root cause not confirmed.)
2. Make the fixture / tests independent from the presence of additional files in the store by retrieving the source filesystem_id and using it to identify the file to delete in the store (preferred).