DataHUB upload of very large data amounts (while local storage is limiting)

[Brilator]

Use-case

• very large data that wants to be uploaded (pushed) as early as possible to the DataHUB
• limited local storage, git by design duplicates data
  • e.g., if raw data on workstation is 5 TB, workstation free storage is 9 TB, git commit on all data in one go would not work

Considerations

• File processing location (local vs. remote)
• Are the files still needed locally? Or are they already processed and it is enough to keep the derived data locally?

Possible solution

1. Add / commit / push each file individually (or in small batches)
2. After push, replace the local ARC with a fresh clone excluding the LFS objects

sequenceDiagram   

    participant LocalData as Local Data folder
    participant LocalARC as Local ARC folder
    participant DataHUB as DataHUB ARC repository

  LocalARC->>DataHUB: Initialize empty ARC

  Note over LocalARC: Loop file-by-file upload process 
  
  loop For each fileX from 1 to n
      LocalData->>LocalARC: Move fileX to ARC
      LocalARC->>LocalARC: git [lfs track + add + commit] fileX
      LocalARC->>DataHUB: git push
      DataHUB->>LocalARC: Option 1: Replace local ARC with LFS-filtered clone
      DataHUB->>LocalARC: Option 2: Replace fileX with LFS pointer (lfs migrate export)
  end

Loading

@j-bauer @TetraW