Checklist for merging release into master

Basic steps:

We use gitflow to manage our branches, along with a devel branch for connecting to bioconductor. This implies all changes are done on the develop branch. When these changes are stable to make a new release, follow the following strategy (note, this assumes pushing to development branch of bioconductor, and not a bug fix on current release in bioconductor):

• Make release branch from develop branch with name of the release corresponding to the version (see Versions below).
• Make final changes/checks on release branch (see below)
• Merge release into develop and master branch using gitflow, and assign tag giving version.
• Change the version of R version in travisCI config file back to match that of master or develop, even if no conflict is created.
• Pull any changes from bioconductor, deal with any possible conflicts in version numbering in DESCRIPTION file.
• Push master into the bioconductor master (see here ).
• In git hub edit the release page with the NEWS information (under releases, go to tags and Edit release notes)
• In develop branch, start a x.y.z-9001 version to match new version number

If this is an update right before Bioconductor release:, Bioconductor will update the version number. So the release page, tag, and so forth should wait until bioconductor updates the version number upon release. After this happens, the steps on the release, etc. should be dealt with.

Version numbering

See here for bioconductor rules for version numbers. If the version is x.y.z, then Bioconductor wants the development version to have y be odd. When they release a new bioconductor (roughly every 6 months) then y gets increased to the even (y+1).

If this update is being submitted right before a bioconductor update and is the last before the bioconductor update then bioconductor will bump up the number. So for all of these instructions make the release branch, the tag and git release page the number that it WILL be in the new bioconductor package so that they match. Of course the DESCRIPTION file is changed by bioconductor, so don't change it in the release/x.y.z DESCRIPTION file until after you port everything into bioconductor. Similarly, if bioconductor is about to bump up the number, you need to make the DESCRIPTION file in develop that adds -9001 to the even version that will be on bioconductor (i.e. x.(y+1).9001) -- the idea being that you release it, it will be bumped up to x.(y+2).0.

If this update is the first since a bioconductor update, then bioconductor will have gone to x.y.z where y is even. So you need to check that it is the 0th version with y odd (i.e. x.(y+1).0).

To Do On Release Branch:

The following checks and changes should be made on a release branch before merging into master or bioconductor:

• If this is a new bioconductor release version, deal with defunct and deprecated functions (see here )
• update NEWS
• bump up version in DESCRIPTION. This should match what should go into bioConductor, not the version bioconductor will bump it up to. You should also check that bioconductor hasn't bumped up the version number in the last update (e.g. from 1.1.xx to 1.2.0). See version section above.
• Check the log of TravisCI from develop branch to review WARNINGS or NOTES (which would not cause a failed result in TravisCI). Particularly check for documentation problems.
• Run document() to make sure .Rmd files are up-to-date.
• If there are any changes in the clustering functions, do check on clusterMany under 'tests/checkClusterMany' to make sure clustering hasn't changed. This requires getting the L5_sumExp.rda file from the git repository using git lfs.

#in the tests/checkClusterMany folder:
nohup RScript clusterManyTest.R releaseCheck

(releaseCheck can be any tag desired). If you get an error about bad restore file magic number in loading L5_sumExp.rda it probably means you only have the tag of that file, and haven't actually pulled it from git lfs.

The output should be in releaseCheck_x.xx.x.Rout (where x.xx.x is the version number). At the top, look for text like the following:

Results for test of 0.99.4 
-------------------
Running clusterMany...done.Current Version: 0.99.4 
User-given tag: releaseCheck 
Compare releaseCheck_0.99.4.txt to fixed version ( fixedClusterManyResult.txt ) :
Are all entries the same?
 Yes 
-------------------

If all entries are not the same, then there is clearly a problem. The results of the clustering with this version can be found in the file releaseCheck_x.xx.x.txt if that is helpful for debugging. Note that the above call uses load_all in devtools, rather than the installed library so as to be current to the existing code.

• Push the above to git repository and make sure no errors in TravisCI (and check warnings and notes again). You will need to first run it with source: true in the travisCI file to download. Make sure to also change so it checks against the devel version of R (since has to be downloaded from scratch anyway since its a new branch...).

To merge into bioc on devel branch (out-of-date):

This material is no longer needed as bioconductor is no longer a svn respository. I have left it here for the information it contains.

Adapted from https://www.bioconductor.org/developers/how-to/git-mirrors/

First you want to clone a new copy somewhere else, so that if (when!) you make mistakes you can just delete it and try again:

git clone [email protected]:epurdom/clusterExperiment.git
cd clusterExperiment
curl -O https://raw.githubusercontent.com/Bioconductor/mirror/master/update_remotes.sh
bash update_remotes.sh
git checkout devel
git pull

This creates a new copy and sets up the connection to the devel (Bioconductor) branch.

The basic steps for updating devel (the logic behind them is explained more after these bare-boned instructions):

• Find the commit that was the merge of the old release into develop, i.e. the release that was used in previous bioconductor update. Call this CCC. One of the parents of CCC will be the commit from the old release, call it BBB. BBB should also match the last commit on the devel branch. If not, you need to add any commits made on the old release branch that are missing from the bioconductor devel branch using the "Simple update" instructions below until BBB is on the devel branch.
• Now you need to cherry-pick CCC into the devel branch. Because it's a merge, you have to find out which parent of CCC is BBB (see "Cherry-picking Merges" below). You can do this by

git show --pretty=raw CCC

The first parent listed is 1 and the second is 2. When you figure out if BBB is 1 or 2 you then cherry pick CCC by

git checkout devel
git cherry-pick -m X CCC

where X is either 1 or 2 as found above.

• We assume that the next commit you want to add after CCC, say DDD is NOT a merge. If it is you should repeat the above, until you get to a non merge commit.
• Now we will rebase the remaining history to make it linear (See the section below "Rebasing to linearize history" for more explanation). Checkout the current release branch that you want to update into bioconductor and create a copy of it

git checkout release/x.x.x
git checkout -b tempRelease release/x.x.x
git rebase DDD

You are likely to get conflicts in files like NEWS or DESCRIPTION. Follow the instructions for fixing the conflicts and keep moving.

• We will move back to the devel branch and cherry pick DDD from the tempRelease branch all the way to the end of tempRelease (Note we say CCC in our cherry pick, because sequences in github do NOT include the first indicated)

git checkout devel
git cherry-pick CCC..tempRelease
# Fix any conflicts that arise, then run
git cherry-pick --continue

You are likely to still get conflicts in files like NEWS or DESCRIPTION. Follow the instructions for fixing the conflicts and keep moving.

• check that that there are no differences with the release branch as possible
• Check that these commands work with no error:

# both of these commands should now work without error:
git svn info
git svn --username=your.username rebase

• Commit to svn:

git svn dcommit

Simple cherry-pick

The basic idea is to take the commits that have been added and cherry pick them to the devel branch (the conduit to the bioconductor svn).

git checkout devel
git pull
git cherry-pick AAA
# Fix any conflicts that arise, then run
git cherry-pick --continue

If all of the commits from AAA to BBB have been linear (no merges) you can grab a series of commits by:

git checkout devel
git pull
git cherry-pick AAA..BBB
# Fix any conflicts that arise, then run
git cherry-pick --continue

Note AAA is the last shared commit between devel and the branch you are cherry-picking from (Otherwise you can do AAA^ to indicate the first non-shared commit). -X theirs can be added to tell cherry-pick to prioritize the other branch in any conflicts (you still may get conflicts if there is a deleting of files).

If there is a conflict (probably in DESCRIPTION) then change the file. Then

git add DESCRIPTION
git cherry-pick --continue

Cherry-picking Merges

However, cherry-pick does not like merges. So if there is a merge, you should not include a merge in the range AAA..BBB given above. Instead, the merge should be cherry-pick'd in its own step, i.e. just that merge commit is cherry-picked. For cherry-picking a merge commit you must indicate in the cherry-pick command which parent you want to use.

So suppose that the result of the merge from release into develop was commit CCC that you want to add. Then we cherry-pick just CCC and indicate the parent of CCC. This tells git which diff we want to use to paste onto devel when we cherry-pick (cherry-pick just takes a diff and adds it to where you are). The syntax for this is:

git cherry-pick -m X CCC

where X is either 1 or 2 and indicates which parent you want to take the diff from. You want the parent to be the commit that matches up with what is on the devel branch. But which is 1 and which is 2? The following command will print out the parents of commit CCC

git show --pretty=raw CCC

It could look something like this:

commit 3575ca98a48c87728c1bf20a4ef073e624387029
tree 5a93d86e4f813d58df0791b5ff77eddd37af516d
parent 27d408084cf10cda86441b60d116967b00b67a88
parent 10cb9ba33198172979044fbd0e26e04da06a0e1f
author Elizabeth Purdom  1477288005 -0700
committer Elizabeth Purdom  1477288005 -0700

    Merge branch 'master' of github.com:epurdom/clusterExperiment into develop

Then 1 refers to the first parent listed and 2 refers to the second one listed.

After you get past a merge, you should then be able to merge a series of commits using the CCC..BBB syntax again if needed.

Note that if the only merge is e.g. the release into develop, and there were not changes in develop since the creation of the release branch, you may get some kind of error about this being an empty commit. Just follow the instructions to get it added.

Rebasing to linearize history

In principle you could slowly walk through the cherry pick one by one and indicate the right parent as above. And manually working through a large merge of branches (e.g. release and develop) is probably the best strategy. However, if there is fairly simple off branches that merged back in (or feature branches), this is really tedious.

We follow this trick from the answer of Daira Hopwood on stackoverflow (http://stackoverflow.com/questions/9229301/git-cherry-pick-says-38c74d-is-a-merge-but-no-m-option-was-given) to fix this problem. Basically you create a temporary branch that is a copy of the release, and then you linearize the history using rebase since the last bioconductor release.

First, it is best to follow the above step to get past that merge of the release with the develop (commit CCC above). So assume you've followed the above and cherry-picked CCC to devel, and now you want to cherry pick the next commit (DDD) all the way up to the end, but this history involves merges in the develop branch.

First we create a temporary branch that will linearize the history from the last commit you cherry-picked into devel (CCC) and the end of the release. The new branch will be tempRelease. If you want DDD through the end of "release/x.x.x", you create a temporary branch where you will linearize the history between DDD and and the end of release

git checkout release/x.x.x
git checkout -b tempRelease release/x.x.x
git rebase DDD

There are likely to be conflicts that need to be resolved. But once you finish it, your history should be completely linear up to DDD.

Now you will cherry-pick those commits to the devel branch.

Because we have linearized the history, this will now be (hopefully!) easy.

git cherry-pick CCC..tempRelease
# Fix any conflicts that arise, then run
git cherry-pick --continue

In practice, you may need to cherry-pick a couple of commits between CCC and DDD to devel before you get to a convenient point DDD where you have a non-merge commit to start your rebase from. (and adjust the cherry-pick command accordingly)

You sometimes get empty commits because the commits have been already added (not sure why, probably misjudge where devel left off). Just follow the instructions to skip them.

Setting up svn the first time:

The first time the repository was linked to svn, the above git code was not used. Instead the following code was used to link up the svn devel branch to the existing master (as suggested by wjiang2 at fhcrc.org):

bash update_remotes.sh
git checkout devel
git rebase	
git log #to get the git-svn starting point commit
git checkout master
git merge -s ours git_svn_starting_point_commit_id #won't do anything but interconnect git and git-svn branches so they are not totally different
git checkout devel
git merge --squash master
git commit -am "comments"
git svn dcommit