The Great Project Restructuring of 2018

[vito]

Context: https://discuss.concourse-ci.org/t/psa-the-great-code-restructing-of-2018/543

• Automate creation of "one big repo"
  • Done: https://github.com/concourse/one-big-repo/tree/maintenance
• Get a proof-of-concept going using Go 1.11 modules
  • Done: https://github.com/concourse/one-big-repo/tree/master
• Ship 4.1.0 4.2.1
• Prep workflow around "one big repo"
  • Design pipeline for new structure
    • Figure out how concourse-bosh-release should be built (solution: auto-bump concourse blob)
  • Create a really nice README.md (finally)
  • Preserve CODE_OF_CONDUCT.md, LICENSE.md, NOTICE.md (update attribution header? contact legal)
  • Create a new CONTRIBUTING.md for the newer (hopefully much much simpler) workflow
  • Create a better issue template
    • Instruct users to open "ops" issues on deployment-specific repo (try to limit concourse/concourse to actual bugs and feature requests)
• Create concourse-bosh-release repo containing the current content of concourse/concourse
• Clean up all stale branches under concourse/concourse
  • Migrate any ones we still need to concourse-bosh-release
• Create "one big repo" based on prior automation, this time "for real"
• Push the "one big repo" as a merge (with distinct parents) to the concourse/concourse master branch.
• Archive atc, fly, go-concourse, tsa, web, testflight, ...
• Restructure "one big repo"; look to other "one big repos" for patterns (e.g. pkg/..., cmd/..., web/...) (long-term)

[jama22]

"blocked" on a release....but @vito will keep working on READMEs and what-not

[aegershman]

I just want to tell you both good luck. We're all counting on you.
(and thank you for your OSS work. we the community genuinely appreciate it 👍you're helping make our lives easier and better)

[vito]

@aegershman Hah, thanks!

---

I just realized we're in a bit of a pickle here. The original plan was to just push -f over the concourse repo, and this is so that we can keep all the existing issues here, since they can't be moved.

But, there are also GitHub releases on the concourse repo, and all of our GitHub releases contain all the artifacts for previous versions of Concourse. If I push -f over master of concourse/concourse, all the old tags will go away, and as a result all existing releases will be turned into draft releases and unpublished. We would have to go back and manually re-add them, but all their tags wouldn't even point to anything useful, because the history is completely new and contains the sum of all the different repos, and nothing related to the release itself. I guess the tags would still be present on the concourse-bosh-release repo if we really want to see the code for a given version.

I could rename the repo instead, but that would in turn move all the existing issues to the concourse-bosh-release repo. This maintains the history much better and feels like a smoother transition, but all the issues would be in the wrong place. :/ And all the existing links between issues would break as soon as we add the one big concourse repo and make the name active again.

So we need to pick our poison: lose all release history but preserve tags on concourse-bosh-release, or rename the repo and break GitHub issue links and end up with all the GitHub issues on the wrong repo. Maybe we could also ask GitHub for help here? :/

[vito]

Hmm, actually we might be able to get away with this. When you push -f over master, that'll blow away master with a whole new history, but all the old tags are still there, keeping the old refs and commit history alive. The GitHub releases also stick around, with their artifacts and everything.

Side note: even with all the old tags keeping the history around, the resulting concourse repo is still pretty small. The "one big repo" with everything stripped down was ~26MB, and the push -fed repo that still contains tags and such is ~31MB. Not bad!

[vito]

Other things I'm thinking about while tackling this:

• We should stop checking in compiled assets like bindata.go and elm.js.min. Repository size is a concern now that we've got everything in here together. As part of the "one-big-repo" creation script I've used BFG to clean up the history, to great effect.
  • We should switch from go-bindata to something that's still maintained. statik looks good.
• (Utilize gdn Guardian binary rather than importing their code directly #2574) For the Linux developer workflow, we should make it easier to go from 0 to a fully working Garden-runC Linux worker.
  • Right now we directly import guardiancmd and embed it in the binary go-flags struct. This is kinda cool but is also tightly coupled and means we have to hoist around all of its binary dependencies. There is a self-contained gdn binary now (which hoists around and auto-wires its own dependencies) - should we just use that instead? (opened Utilize gdn Guardian binary rather than importing their code directly #2574 for this)
• (Independently version all "core" resource types #2576) The build process for the concourse binary is fairly obtuse and depends on the BOSH release for embedding resource packages directly into the binary. I think we should probably change this, but I'm not sure what to do yet. It probably makes more sense to just ship the binary alongside the resources rather than embedding it.

[vito]

More questions as I discover all the ways the BOSH release workflow has worked its tendrils into our process. For fear of getting trapped in analysis paralysis, I'm at least going to write them down before coming up with answers. I guess I'll just keep editing this.

• (Use concourse binary in BOSH release #2575) When and how will we build Elm and other static assets in to the binary/BOSH release, if we're not checking it in?
• (Use concourse binary in BOSH release #2575) How do we ensure the BOSH release can build without needing to fetch Go dependencies? (Can we vendor prior to building the release and convince the bosh CLI that the tree isn't dirty somehow?)
• (Use concourse binary in BOSH release #2575) Would all of the above be easier if we just had the BOSH release be a wrapper for the concourse binary? Would the loss in flexibility be worth it? What other benefits might there be?
• (Independently version all "core" resource types #2576) Is it time to just start versioning and shipping the core resources properly?
  • Is this also a good time to 'trim down' the core set of supported resource types?
• How do we cut over to this process without stopping the world and making it impossible to deliver for a while?
  • I don't think you do.

[vito]

I've started switching everything over to packr: https://github.com/gobuffalo/packr

It's a lot better suited than the previously-mentioned statik package because it lets us have an arbitrary number of statically embedded things, rather than just the one single package that statik provides. It also works really well for local development; it'll just read from disk, and then in our pipeline we just run packr to snapshot the state and embed it right in the binary. Neat!

[jchesterpivotal]

Force-push (no, really) the "one big repo" over concourse/concourse master branch.

It's actually possible to merge repos in a way that preserves their independent histories using --allow-unrelated-histories. The outcome is that it looks like a set of merge patches from branches representing each of the repositories.

(Maybe this is what you already intend, I'm just an anxious person)