Remove binary blobs from history

[aozarov]

Following the steps described in issue #11

Step 1. Remove src/test/resources/gcd-v1beta2-rev1-2.1.1.zip from index.

git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' --tag-name-filter cat -- --all

Step 2. Remove src/test/resources/gcd-head.zip from index.

git filter-branch --force --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-head.zip' --tag-name-filter cat -- --all

Step 3. Ignore those files from now on, make a new commit.

echo 'src/test/resources/gcd-head.zip' >> .gitignore
echo 'src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' >> .gitignore
git commit -am 'Remove files'

Step 4. Garbage collect outdated blobs and force push back to your fork.

(WARNING THIS WILL REWRITE HISTORY IN YOUR FORK)

git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
git push origin --force --all

[jgeewax]

@ryanseys : Any chance you can double check this too? It's a big scary change that we want to make sure we don't have to do again........

[ryanseys]

Because it rewrites history we can't merge this in as a PR, it has to be force pushed to the repo and there's really no way for me to "double check" this. I'm not sure what I would do to double check it considering all the SHAs have changed, hence the monstrosity of a PR this is. Any thoughts?

[aozarov]

Something strange.

When I clone gcloud-java from GoogleCloudPlatform and from aozarov (where I applied the pruning)
I don't see a difference in size (actually the clone is slightly bigger...)

ozarov /tmp $ git clone https://github.com/GoogleCloudPlatform/gcloud-java/
Cloning into 'gcloud-java'...
remote: Counting objects: 71093, done.
remote: Total 71093 (delta 0), reused 0 (delta 0), pack-reused 71093
Receiving objects: 100% (71093/71093), 225.40 MiB | 3.56 MiB/s, done.
Resolving deltas: 100% (66008/66008), done.

ozarov /tmp $ git clone https://github.com/aozarov/gcloud-java/
Cloning into 'gcloud-java'...
remote: Counting objects: 71176, done.
remote: Total 71176 (delta 0), reused 0 (delta 0), pack-reused 71176
Receiving objects: 100% (71176/71176), 225.40 MiB | 3.33 MiB/s, done.
Resolving deltas: 100% (66043/66043), done.

Does it make sense? Could it be that the clone has more branches to be the reason?

Also, I am not sure why the Pull request suggests 200 files change (Last time I tried it,

Before, using a different procedure I mentioned in issue #11 there were many commits to merge but no file differences (see see pull request #73).
A local diff between the 2 clones and also between the 2 zip files downloaded from both sites suggest only .gitingore as a difference (so it is probably OK, but just want to make sure).

Maybe a way to verify it is to duplicate the repository and then apply the procedure on a clone of it, push the change and see if it works?

[jgeewax]

@ryanseys mentioned in a previous post that he had to delete the old repository to actually get rid of things, but potentially we could simple prune old objects after a git push -f. Did you force push this to your fork?

As Ryan said, the PR won't work here, so this is just a debate on the change until we ultimately apply it forcefully.

[aozarov]

Yes, I applied all the steps described by @ryanseys in issue #11. I also mentioned the steps I applied above, which include the git push origin --force --all. Is that what you are referring to?

Oh, are you (and Ryan) saying that once agreed I will need to apply the same steps directly on the GCP repository instead of merging it via a push request from a fork? If so, I still don't understand why this fork does not seem to be smaller...

[ryanseys]

Yeah, apparently they only run garbage collection on the repo a maximum of once per day on a repo and it's triggered by a push so you may have to wait a couple days and push a couple dummy commits in order to see a difference in clone size once this GC runs.

Source: https://twitter.com/GitHubHelp/status/387926738161774592