Overall improvements to workflow parallelization

[nfahlgren]

This is an Epic to help organize the overall efforts to improve PlantCV workflow parallelization. There are three main components to workflow parallelization:

• Metadata processing/handling
• Workflow handling
• Parallelization

Various improvements have been suggested for both areas. We will attempt to subdivide these into discreet issues as much as possible, but many improvements will be inherently interlinked. Areas of improvement are (in no particular order):

• Update input parameter names for consistency update flags in to be more consistent between a workflow and plantcv-workflow.py #470
• Utilize configuration files instead of lengthy command-line options update flags in to be more consistent between a workflow and plantcv-workflow.py #470
• Image grouping or aggregation for data series (e.g. multi-frame image sets, co-processing, etc.) parallel pipeline for image series #423
• Adding metadata directly in workflows Is it possible to manually add metadata to a non-workflow job? #490
• Improve/expand valid metadata terms
• Allow multiple metadata terms in the metadata filter Valid metadata in plantcv-workflow.py cannot be duplicated #385 'match' multiple values of same metadata field #461
• New batch processing system(s) New function: Add queuing system/batch processing option #28
• Store timestamps in ISO 8601 format Store timestamps in ISO 8601 format #474

Will update if I missed anything or plans change.

[dschneiderch]

@nfahlgren this professional isolation moment seems like a good opportunity to work on these. i'd be particularly interested in working on the grouping/mapping functionality although I think most of these issues are inherently linked.
there was some discussion at Phenome about how to tackle the multiple image sets and we tentatively concluded that instead of trying to wrangle the plantcv-parallel.py to accept multiple images, we could write a preprocessing script that bundled groups of images together based on metadata and a user-provided config file. A potential bundle could be a list of filenames, a multiframe tiff, or a zip file.

In #423 @nfahlgren said he had basic functionality for grouping images by metadata (replicating the --coprocess flag). If you upload this branch we could discuss what/where needs to happen to move this forward.

[dschneiderch]

hi guys, wondering if there is any movement towards tackling these improvements now that pcv 4.0 is only one release away :)
I noticed someone opened a timeseries submodule #587 which seems like it might relate to the multiframe functionality we need for psII images. I would be happy to help

[nfahlgren]

Hi @dschneiderch, this kind of fell off to the back burner the last few months. We just had a group discussion yesterday about some needs for the timeseries submodule related to this issue, so definitely related.

The paper still needs quite a bit of writing, but our thinking is that we could move onto the 4.x series soon and closeout 3.x. There are some outstanding issues like this one where it's tempting to hold it over to 4.x because some of the other big changes we want to make in 4.x could make tackling this issue easier. The timeseries subpackage is something we are already for sure holding over to 4.x.

We don't want to hold up progress though, so we're going to make a 4.x branch where we can have pull requests go for non-3.x features. We will keep 4.x rebased on 3.x so that we don't create a big point of divergence.