Define ruleset for lfs/non-lfs files on datahub

[Freymaurer]

• MUST NOT be lfs isa.*.xlsx
• MUST be lfs **/dataset/**
• files above 25mb size MUST be lfs. For this restriction there is no generic pattern. This must be implemented on tool side.

The patterns, which are possible will be added to a reference .gitattribute file via ARCtrl by @HLWeil (link be added below)

These restrictions will be checked server side on the datahub to enforce them.

@HLWeil @Brilator @TetraW @j-bauer

[TetraW]

As mentioned by @Brilator in this discussion, we should probably include /runs/ to avoid leaving many small automatically produced files untracked by LFS.

[Brilator]

Not sure, if these rules should be "defined" on knowledge base. Also not sure where else (e.g. datahub repo?), since ARC spec does not handle git.

• add guide: set git config lfs threshold
• adapt guide: set arc commander config lfs threshold https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arc-commander/lfs/
• adapt guide: handling large files, with clear ruleset (25MB): https://nfdi4plants.github.io/nfdi4plants.knowledgebase/git/git-lfs:

[Brilator]

As mentioned by @Brilator in this discussion, we should probably include /runs/ to avoid leaving many small automatically produced files untracked by LFS.

the run.yml, run.cwl should IMO be whitelisted just like ISA files

[TetraW]

I have created a .gitattributes file in which I tried to take into account all the rules discussed here and also checked it again against the ARC specification. Feel free to suggest changes here or directly under the file.

[Brilator]

So you can LFS-whitelist files by patterns in .gitattributes?
I understood that this impossible.

@Thyra

[Thyra]

@Brilator I don't know, I only meant that the LFS-whitelist (called blacklist there) in ARCitect (additionally) uses hard-coded patterns which are not present in the .gitattributes file (e.g. to exclude isa.* files).

( https://github.com/nfdi4plants/ARCitect/blob/f909952d5a79789f71babd8b006d68f610dc82b7/packages/renderer/src/GitService.ts#L24 )

[TetraW]

@Brilator In general, this .gitattributes file should work. I tested it with two files, runs/test_run/data.bin and runs/test_run/run.yml, in a test repository on the DataHUB, and the LFS behavior was as expected.

I have not cross-checked this file with the one used by ARCitect. One important detail: for .gitattributes, if multiple patterns match the same path, the last matching line wins for that path, so the order of rules in the file matters.

[Brilator]

I have not cross-checked this file with the one used by ARCitect. One important detail: for .gitattributes, if multiple patterns match the same path, the last matching line wins for that path, so the order of rules in the file matters.

Ok, that's quite important. So no matter how good of a .gitattributes an ARC starts with, the tooling (ARC commander, ARCitect) must still check and prevent that those files will be LFS-tracked.

[Brilator]

this .gitattributes file should work. I tested it with two files, runs/test_run/data.bin and runs/test_run/run.yml, in a test repository on the DataHUB, and the LFS behavior was as expected.

What do you mean by "as expected"? My expectation would be, that trying to git lfs track runs/test_run/run.yml should prompt a warning. Instead this will just list that file last line and override the previous rules

[TetraW]

What do you mean by "as expected"? My expectation would be, that trying to git lfs track runs/test_run/run.yml should prompt a warning. Instead this will just list that file last line and override the previous rules

I only meant that the mentioned files got LFS tracked (or not) as I intended with the posted .gitattributes file. Sorry, I should have been more specific about that.

I agree that Git handles this in a not very intuitive way with this “last rule wins” approach.

I am sorry, but I currently do not have a better solution for how to manage this directly with Git.

One idea might be to check or enforce a certain .gitattributes layout via server-side hooks that validates the file on push, as we plan to for the single file size limit. In theory this could catch conflicting or “forbidden” patterns and reject the push, but I have not tried this myself, and I am not sure if the added complexity and maintenance effort is worth it. Since ARCitect and other tools would then also need to properly handle these rejections.