[Feature Request] Warn users when they upload large files via lfs without access token

[eik-dahms]

When a user uploads a lot files or large files via lfs and it takes more then two hours the push will fail. This is due to the invalidation of the access token/ not updating it in the push command.

One could set a size limit where the user is warned if the don't use an access token.

[Thyra]

File size is probably not the best predictor because the transfer speed of both the local computer and the DataHUB can be quite inconsistent. But maybe at least a clear error message or some more graceful handling by tracking the upload time inside the ARCitect? I've had users be confused about this as well before.

[eik-dahms]

that is true, but I am pretty sure if my users upload 1TB or more , independent of how many files that might be, that that will take more than two hours. Right now we still have a situation where some users have to use VPN and the upload speed is like 2MB/s.

I think there are two things that could be done here.

Estimate upload time

Maybe there is a commandline argument such that git lfs directly shows the estimated time. Otherwise git lfs looks somthing like this usually

Uploading LFS objects: 100% (926/926), 1.1 TB | 202 MB/s, done.

from this one could extract the upload speed additionally with git lfs status one can get that files that will be pushed. Then file sizes + upload speed can be used to estimate upload time.

Ensure correct token is used and catch authentification errors

From my understanding (please correct me if I am wrong.) the push command is executed once with the token baked into the https url and the main culprit in longer uploads is that original token is not valid anymore after 2hours latest. Even if the token has been updated by reauthentification by the user it will not be updated in the running git push process.

To improve the whole process the following things could be done.:
As already stated in #499 once the user has authentificated the token can be refreshed in the background. Meaning, if that would be implementet, no user interaction is necessary anymore.

Further we can separate lfs upload and the actual push. This would facilitate the control flow and error handling a bit easier. It would look roughly like this in bash:

# upload lfs 
git lfs push --all <remote> <branch>

# git push
git push

I am not a 100% sure but i think the lfs upload also always needs a proper token.
So i would wrap the lfs push in try/catch such that one can react to authentification errors and give it a retry with the new token. Might even be better to put it in a loop which runs as long as not all lfs files are pushed and breaks after one not succseful retry with a new token.

after that we can start the actual push which should then always use the latest token and should not be taking very long anyways.

Lastly it might be worth consider token handling in general - as baking it into the url is possible - but not good practice/ can be security issue. I am not an expert in TS so I am not sure if this could be managed via ENV or with some additional script that can be given to "GIT_ASKPASS"

[JonasLukasczyk]

I would argue for enforcing the use of tokens, i.e., no token no push