Skip to content

Commit 8181280

Browse files
ndimidukthangTang
ndimiduk
authored and
thangTang
committed
HBASE-22853 Git/Jira Release Audit Tool (apache#1088)
* HBASE-22853 Git/Jira Release Audit Tool This is an application for performing an audit between the histories on our git branches and the `fixVersion` field set on issues in JIRA. It does this by building a Sqlite database from the commits found on each git branch, identifying Jira IDs and release tags, and then requesting information about those issues from Jira. Once both sources have been collected, queries can be performed against the database to look for discrepancies between the sources of truth (and, possibly, bugs in this script). Signed-off-by: Sean Busbey <busbey@apache.org> Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
1 parent ecbaa8f commit 8181280

6 files changed

Lines changed: 1003 additions & 0 deletions

File tree

‎.editorconfig‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -691,3 +691,7 @@ ij_ruby_spaces_around_range_operators = false
691691
ij_ruby_spaces_around_relational_operators = true
692692
ij_ruby_spaces_within_array_initializer_braces = true
693693
ij_ruby_spaces_within_braces = false
694+
695+
[*.py]
696+
indent_size = 4
697+
tab_width = 4

‎dev-support/git-jira-release-audit/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
*.db
2+
*.log
3+
*.svg
4+
venv

‎dev-support/git-jira-release-audit/README.md‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Git / JIRA Release Audit
21+
22+
This is an application for performing an audit between the histories on our git
23+
branches and the `fixVersion` field set on issues in JIRA. It does this by
24+
building a Sqlite database from the commits found on each git branch,
25+
identifying Jira IDs and release tags, and then requesting information about
26+
those issues from Jira. Once both sources have been collected, queries can be
27+
performed against the database to look for discrepancies between the sources of
28+
truth (and, possibly, bugs in this script).
29+
30+
## Setup
31+
32+
The system prerequisites are Python3 with VirtualEnv available and Sqlite. Also,
33+
you'll need the content of this directory and a local checkout of git repository.
34+
35+
Build a VirtualEnv with the script's dependencies with:
36+
37+
```shell script
38+
$ python3 --version
39+
Python 3.7.6
40+
$ python3 -m venv ./venv
41+
$ ./venv/bin/pip install -r ./requirements.txt
42+
...
43+
Successfully installed...
44+
```
45+
46+
## Usage
47+
48+
The tool provides basic help docs.
49+
50+
```shell script
51+
$ ./venv/bin/python ./git_jira_release_audit.py --help
52+
usage: git_jira_release_audit.py [-h] [--db-path DB_PATH]
53+
[--git-repo-path GIT_REPO_PATH]
54+
[--remote-name REMOTE_NAME]
55+
[--development-branch DEVELOPMENT_BRANCH]
56+
[--development-branch-fix-version DEVELOPMENT_BRANCH_FIX_VERSION]
57+
[--release-line-regexp RELEASE_LINE_REGEXP]
58+
[--fallback-actions-path FALLBACK_ACTIONS_PATH]
59+
[--jira-url JIRA_URL] --branch-1-fix-version
60+
BRANCH_1_FIX_VERSION --branch-2-fix-version
61+
BRANCH_2_FIX_VERSION
62+
63+
optional arguments:
64+
-h, --help show this help message and exit
65+
--db-path DB_PATH Path to the database file, or leave unspecified for a
66+
transient db.
67+
--git-repo-path GIT_REPO_PATH
68+
Path to the git repo, or leave unspecified to infer
69+
from the current file's path.
70+
--remote-name REMOTE_NAME
71+
The name of the git remote to use when identifying
72+
branches.
73+
--development-branch DEVELOPMENT_BRANCH
74+
The name of the branch from which all release lines
75+
originate.
76+
--development-branch-fix-version DEVELOPMENT_BRANCH_FIX_VERSION
77+
The Jira fixVersion used to indicate an issue is
78+
committed to the development branch.
79+
--release-line-regexp RELEASE_LINE_REGEXP
80+
A regexp used to identify release lines.
81+
--fallback-actions-path FALLBACK_ACTIONS_PATH
82+
Path to a file containing _DB.Actions applicable to specific git shas.
83+
--jira-url JIRA_URL A URL locating the target JIRA instance.
84+
--branch-1-fix-version BRANCH_1_FIX_VERSION
85+
The Jira fixVersion used to indicate an issue is
86+
committed to the specified release line branch
87+
--branch-2-fix-version BRANCH_2_FIX_VERSION
88+
The Jira fixVersion used to indicate an issue is
89+
committed to the specified release line branch
90+
```
91+
92+
Example Run:
93+
94+
```shell script
95+
$ ./venv/bin/python3 ./git_jira_release_audit.py \
96+
--db-path=audit.db \
97+
--remote-name=apache-rw \
98+
--development-branch-fix-version=3.0.0 \
99+
--branch-1-fix-version=1.5.0 \
100+
--branch-2-fix-version=2.3.0
101+
INFO:root:apache-rw/branch-1 has 4046 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
102+
INFO:root:apache-rw/branch-1.0 has 1433 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
103+
INFO:root:apache-rw/branch-1.1 has 2111 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
104+
INFO:root:apache-rw/branch-1.2 has 2738 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
105+
INFO:root:apache-rw/branch-1.3 has 3287 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
106+
INFO:root:apache-rw/branch-1.4 has 3912 commits since its origin at 0167558eb31ff48308d592ef70b6d005ba6d21fb.
107+
INFO:root:apache-rw/branch-2 has 3080 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
108+
INFO:root:apache-rw/branch-2.0 has 2194 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
109+
INFO:root:apache-rw/branch-2.1 has 2705 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
110+
INFO:root:apache-rw/branch-2.2 has 2927 commits since its origin at 0d0c330401ade938bf934aafd79ec23705edcc60.
111+
INFO:root:retrieving 5653 jira_ids from the issue tracker
112+
113+
apache-rw/branch-1 100%|██████████████████████████████████████████████████████| 4046/4046 [08:23<00:00, 8.04 commit/s]
114+
apache-rw/branch-1.0 100%|████████████████████████████████████████████████████| 1433/1433 [03:49<00:00, 6.26 commit/s]
115+
apache-rw/branch-1.1 100%|████████████████████████████████████████████████████| 2111/2111 [05:16<00:00, 6.68 commit/s]
116+
apache-rw/branch-1.2 100%|████████████████████████████████████████████████████| 2738/2738 [06:26<00:00, 7.10 commit/s]
117+
apache-rw/branch-1.3 100%|████████████████████████████████████████████████████| 3287/3287 [07:21<00:00, 7.46 commit/s]
118+
apache-rw/branch-1.4 100%|████████████████████████████████████████████████████| 3912/3912 [08:08<00:00, 8.02 commit/s]
119+
apache-rw/branch-2 100%|█████████████████████████████████████████████████████| 3080/3080 [03:29<00:00, 14.74 commit/s]
120+
apache-rw/branch-2.0 100%|████████████████████████████████████████████████████| 2194/2194 [04:56<00:00, 7.42 commit/s]
121+
apache-rw/branch-2.1 100%|███████████████████████████████████████████████████| 2705/2705 [03:17<00:00, 13.75 commit/s]
122+
apache-rw/branch-2.2 100%|███████████████████████████████████████████████████| 2927/2927 [03:28<00:00, 14.09 commit/s]
123+
fetch from Jira 100%|█████████████████████████████████████████████████████████| 5653/5653 [00:58<00:00, 98.29 issue/s]
124+
```
125+
126+
With a populated database, query with sqlite:
127+
128+
```shell script
129+
$ sqlite3 audit.db
130+
SQLite version 3.24.0 2018-06-04 14:10:15
131+
Enter ".help" for usage hints.
132+
sqlite> -- count the number of distinct commits on a release branch
133+
sqlite> select count(distinct jira_id), branch from git_commits group by branch;
134+
3406|apache-rw/branch-1
135+
1189|apache-rw/branch-1.0
136+
1728|apache-rw/branch-1.1
137+
2289|apache-rw/branch-1.2
138+
2779|apache-rw/branch-1.3
139+
3277|apache-rw/branch-1.4
140+
2666|apache-rw/branch-2
141+
1809|apache-rw/branch-2.0
142+
2289|apache-rw/branch-2.1
143+
2511|apache-rw/branch-2.2
144+
145+
sqlite> -- count the number of issues that will be in 2.3.0 that have not been released on any earlier
146+
sqlite> -- version.
147+
sqlite> select count(1) from (
148+
select distinct jira_id from git_commits where branch = 'apache-rw/branch-2' except
149+
select distinct jira_id from git_commits where branch in
150+
('apache-rw/branch-2.0', 'apache-rw/branch-2.1', 'apache-rw/branch-2.2'));
151+
169
152+
153+
sqlite> -- find the issues for which the git commit record and JIRA fixVersion disagree
154+
sqlite> select g.jira_id, g.git_tag, j.fix_version
155+
from git_commits g
156+
inner join jira_versions j
157+
on g.jira_id = j.jira_id
158+
and g.branch = 'apache-rw/branch-2.2'
159+
and g.git_tag is not null
160+
and j.fix_version like '2.2.%'
161+
and g.git_tag != j.fix_version;
162+
HBASE-22941|2.2.2|2.2.1
163+
164+
sqlite> -- show jira non-1.x fixVersions for all issues on branch-2 but not on any
165+
sqlite> -- branch-2.x release branch; i.e., issues that are missing a fixVersion or
166+
sqlite> -- are marked for a release other than (3.0.0, 2.3.0)
167+
sqlite> select g.jira_id, j.fix_version
168+
from (
169+
select distinct jira_id from git_commits where branch = 'apache-rw/branch-2' except
170+
select distinct jira_id from git_commits where branch in
171+
(select distinct branch from git_commits where branch like 'apache-rw/branch-2.%')) g
172+
left join jira_versions j
173+
on g.jira_id = j.jira_id
174+
and j.fix_version not like '1.%'
175+
where (
176+
j.fix_version is null
177+
OR j.fix_version not in ('3.0.0', '2.3.0'))
178+
order by g.jira_id desc;
179+
HBASE-23683|2.2.4
180+
HBASE-23032|connector-1.0.1
181+
HBASE-23032|hbase-filesystem-1.0.0-alpha2
182+
HBASE-22405|2.2.0
183+
HBASE-22360|2.2.0
184+
HBASE-22321|
185+
HBASE-22283|2.2.0
186+
```

0 commit comments

Comments
 (0)