Add in-repo file checksum and integrity check

[neomantra]

I'm looking at adopting this and wanted to track file checksums. Does it make sense to optionally include the sha1 in the .git-remote-files file?

--save could bake those, or perhaps --save-checksums.

Then you can include a verify command that verifies that value with the on-disk value without needing to consult the external repo. I want to track that drift more specifically.

Does this feature make sense? I'm happy to help implement a PR

[andrewmcwatters]

Edit: This sounds like a great idea. Let's do it! I think maybe the command should do this by default, and update the .git-remote-files manifest with a new integrity value per-entry? What are your thoughts?

It appears that there is an attempt to move git from sha-1 to sha-256 for its hash function, and I've seen other integrity implementations in other places utilize sha384-... and sha512-....

If git itself is undergoing a hash function transition to sha-256, maybe it makes sense for .git-remote-files to use the same hash function.

[neomantra]

Sure, sha-256 makes sense. If we prefix it (sha256:), we can make it dynamic with a default of sha-256, but maybe we don't need the complexity yet.

I was feeling weird about keeping the checksums as it is already in the tree, but really that's for repo integrity and this new file maintains the integrity of the local/remote+snapshot relationship.

As I thought more about this, I thought it might make sense to keep two files, in order to make it more clear what is locally stored from remote in the repo:

1. the current .git-remote-files which specifies the configuration, ie files and globs, branches/commits
2. a .git-remote-checksums which keeps a list of actual files sync'ed and the most recent checksums; perhaps commit sha like --save can go there as well.

[andrewmcwatters]

Oh that's a cool idea keeping the checksums stored like that. It reminds me of other ecosystems which have lockfiles!

[khusmann]

I definitely like the checksum idea, especially in the case where fetch-file is in branch-mode rather than in commit-mode.

Otherwise, as a user downloading a fresh repo I don't know what version of the file is currently being used.

Actually do we want file checksum here, or simply always record the commit hash of the downloaded file (in addition to the branch we're pulling from?) In this case, instead of git fetch-file verify you could have a git fetch-file diff where you could see any local changes from the original fetched version.

[andrewmcwatters]

Yeah, I recently ran into a case like this where at Planimeter, we're starting to use this in a small 3D project (https://github.com/Planimeter/game-engine-3d).

Ryan hasn't cut a new release of physfs yet, so I just git fetch-file add the whole repository as a vendored dependency (git fetch-file add https://github.com/icculus/physfs.git "**" lib/physfs-main), because we can't build the project without updating the downstream CMakeLists.txt which only exists in main.

But we also had to patch his CMakeLists.txt for a macOS distribution, so we're:

1. Using git fetch-file for submodule-like usage
2. Making a change in a single file in that subset of "**" (the whole repository)

So, it seems like maybe we should always have some reference of the commit, regardless of branch-tracking or if we're "detached" and using a specific commit.

But we should also deal with the case of having local changes, too, and maybe loop in some sort of "merge conflict" of sorts?

I need to dwell on this. It feels like this will be a very common case for people. Thoughts are very much welcome.

[khusmann]

So, it seems like maybe we should always have some reference of the commit, regardless of branch-tracking or if we're "detached" and using a specific commit.

Exactly

But we should also deal with the case of having local changes, too, and maybe loop in some sort of "merge conflict" of sorts?

That sounds like a lot of potential to get pretty complicated -- I'd be content with the following rules:

• git fetch-file pull would pull the commit referenced in .git-remote-files (and -f would overwrite any local changes)
• git fetch-file diff that would give me a diff of the file's local deviations from the commit it came from
• git fetch-file update would update the commit referenced in .git-remote-files to the latest commit on the branch it was tracking

Or something like that...

[andrewmcwatters]

That sounds like a lot of potential to get pretty complicated

Sigh, yeah. It would be really nice to avoid this, if we can.

Love pull -f, love diff, and yeah we need update.

For now, maybe pull (without -f) just warns of local changes, but leaves it to the user to resolve on their own so we don't introduce our own merge conflict system that looks like git fetch or git pull. That's a can of worms and creates a much larger number of questions.

diff feels less complicated, and maybe we can respect things like whitespace configurations in diffs or something. Not sure yet.

update and pull/pull --save feel like they do similar things, and so I need to work out in my mind what the pinned/detached state resolution looks like in .git-remote-files.

[khusmann]

update and pull/pull --save feel like they do similar things, and so I need to work out in my mind what the pinned/detached state resolution looks like in .git-remote-files.

In my thinking the difference would be that update simply updates the pinned commit hash to the latest on the branch and never downloads files; whereas pull always downloads a file. So update for a detatched state would be a no-op because there was no upstream branch to update to.

Example: in a tracked branch we'd have a .git-remote-files like this:

[file "README.md"]
repo = https://github.com/andrewmcwattersandco/git-fetch-file.git
ref = main
commit = 66d072a

git fetch-file pull would download README.md with commit 66d072a
git fetch-file update would check out branch main, determine the latest commit, then replace commit = with the latest commit hash. No files would be downloaded; the user would then have to git fetch-file pull -f to download the new files. (but first could inspect incoming differences via git fetch-file diff

For a untracked / headless commit we'd have a .git-remote-files like this (no ref = ):

[file "README.md"]
repo = https://github.com/andrewmcwattersandco/git-fetch-file.git
commit = 66d072a

In this case, git fetch-file pull would download README.md with commit 66d072a, as before.

But git fetch-file update would be a no-op, because there's no branch that its tracking. (Although perhaps you could manually set the version via get fetch-file update README.md <commit hash>?)

[andrewmcwatters]

commit only sounds right. This allows us to avoid additional keys for pinning.

I hope I haven't made pull --save messy conceptually, but I fear I have. I also worry about some of these subcommands potentially confusing users from other ecosystems. npm install --save, and npm update for example.

I don't necessarily want to mimic other language ecosystems since we're creating a git-specific and arguably generic solution for cases where using package managers won't help you, but since we're sort of adjacent to that problem space, I'd like to keep the cognitive load low for users.

update feels right, in other ecosystems or even Linux distributions (I don't want to digress too much here, though,) it is an effective operation rather than a no-op, but I think the right thing to do here is keep pull doing the actual pulling. It's the command that git users are already familiar with.

I'll keep thinking about this.

[khusmann]

Ohhhh I didn't realize pull --save was an existing command that updated the commit hashes -- yeah that would get conceptually messy with the scheme I outlined above. pull --update would make more sense to me in that context (equivalent to a update then pull).

--save is just a bit ambiguous because it makes me wonder if I'm force-saving the file (like pull -f), saving a new commit hash (with or without downloading the file?), or something else. But maybe that's because I'm already stuck thinking in the framework I proposed above haha.

Anyway, yeah all good things to stew on. Thanks again for your work on this! I was in a situation wishing for exactly a tool like this a literal week before I saw it on HN, so you caught me at a perfect time.

[khusmann]

update feels right, in other ecosystems or even Linux distributions (I don't want to digress too much here, though,) it is an effective operation rather than a no-op,

Side note -- I think this behavior still fits / transfers well in the detached / pinned situation -- in other these other ecosystems update is often also no-op if a package is pinned.

[andrewmcwatters]

Yep, I'll try to iron this out. I'm a little rough around the edges on this implementation. I think maybe git fetch-file add needs to behave maybe a little bit closer to git-checkout in that it might make more sense to use -b, --branch and --detach for specific commits, which is effectively pinning. We'll keep --commit so we're not breaking anything.

I think you're right that there's still a need for an update here. I think that's our solution to the HEAD detached at.../pinned commit equivalent.