factoring out chunks of this repo

[nsheff]

The goal is to have several repos, each representing a server, which can be taken over by a community that governs the contents of that server. Generic components apply to all servers and should not live in every repo.

Right now this repo really has a several things:

• asset_pep/ -- a PEP describing the assets for this server
• pipeline_interfaces/ -- looper interfaces for downloading, building, and archiving . This is the connection between the PEP and refgenie.
• config/... -- the config file representation of that PEP (output of refgenie runs)
• task_defs/ -- the AWS image descriptions for autodeploy
• .github/ -- github actions defined for automatic deploy

Some of these are specific to this server instance, others are generic.

specific: asset_pep, config.
generic: pipeline_interfaces, task_defs, .github

We could:

• factor out taskdefs and .github into a generic github action that could be called from multiple repos.
• move pipeline interfaces into a separate repo that provides everything needed for the download/build/archive steps.

[nsheff]

Since the .github workflow is using more than one action, it's non-trivial to extract those into a separate action. Some day, it may be possible to construct an action from multiple other actions (I think GitHub is working on this), and we can revisit that then.

It may be possible to factor out the taskdefs, and then just check that out.

I believe refactoring the pifaces would also be possible.

[stolarczyk]

It may be possible to factor out the taskdefs, and then just check that out.

the task definitions are not identical, actually. The hostPorts are variable:

[mstolarczyk@MichalsMBP code]: diff refgenomes.databio.org/task_defs/primary.json rg.databio.org/task_defs/dev1.json 
12c12
<           "hostPort": 85,
---
>           "hostPort": 86,
[mstolarczyk@MichalsMBP code]: diff refgenomes.databio.org/task_defs/primary.json plantref/task_defs/plant.json 
12c12
<           "hostPort": 85,
---
>           "hostPort": 84,

There's no way to dynamically populate values in the task definitions in amazon-ecs-render-task-definition action provided by AWS: aws-actions/amazon-ecs-render-task-definition#20.

We could create a populate-refgenieserver-task_def action that would store the action template, take hostPort as the parameter and populate, but that's overkill in my opinion.

[nsheff]

yeah that's annoying.