Get Codecov to run on PRs from outside the project

[matthewfeickert]

Description

At the moment we have it so that Codecov will only run and report coverage on PRs made from branches on the main project, which basically limits it to only being useful to the core devs.

pyhf/.github/workflows/ci.yml

Lines 39 to 45 in 4e0f92e

	- name: Report core project coverage with Codecov
	if: github.event_name == 'push' && matrix.python-version == 3.8 && matrix.os == 'ubuntu-latest'
	uses: codecov/codecov-action@v1
	with:
	token: ${{ secrets.CODECOV_TOKEN }}
	file: ./coverage.xml
	flags: unittests

To make it a better experience for other contributors and to avoid having to make a shadow branch on the main project to trigger codecov we should figure out a way to get Codecov to run on all PRs in a way that doesn't leave the project open to security leaks.

Recent example PR where this happened: PR #1556

[matthewfeickert]

Note to self: Check if there is a way to do multiple if statement in GitHub Actions or something like

(github.event_name == 'push' && matrix.python-version == 3.8 && matrix.os == 'ubuntu-latest') || (github.event_name == 'push' && matrix.python-version == 3.8 && matrix.os == 'ubuntu-latest' && github.repository != 'scikit-hep/pyhf')

to also pick up PRs from outside the main project. @henryiii have you ever done something like this before?

[matthewfeickert]

Maybe https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/ provides a relevant solution with pull_request_target.

[matthewfeickert]

Apparently, no, given that https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ makes it clear that using pull_request_target with untrusted PRs is very bad for security.

[matthewfeickert]

Okay, so seems things are still not quite working as intended, as PR #1613 is triggering the maintainer logic

pyhf/.github/workflows/ci.yml

Lines 52 to 59 in e47b1cf

	- name: Report core project coverage with Codecov (maintainers)
	# Run on all events except schedule if PR comes from maintainers
	if: github.repository == 'scikit-hep/pyhf' && github.event_name != 'schedule' && matrix.python-version == 3.9 && matrix.os == 'ubuntu-latest'
	uses: codecov/codecov-action@v1
	with:
	token: ${{ secrets.CODECOV_TOKEN }}
	files: ./coverage.xml
	flags: unittests

instead of the contributor logic

pyhf/.github/workflows/ci.yml

Lines 61 to 67 in e47b1cf

	- name: Report core project coverage with Codecov (contributors)
	# Run on pull_request events if PR comes from contributor fork
	if: github.repository != 'scikit-hep/pyhf' && github.event_name == 'pull_request' && matrix.python-version == 3.9 && matrix.os == 'ubuntu-latest'
	uses: codecov/codecov-action@v2
	with:
	files: ./coverage.xml
	flags: unittests

:/

This "makes sense" to me now as pull_request events from forks

When you create a pull request from a forked repository to the base repository, GitHub sends the pull_request event to the base repository and no pull request events occur on the forked repository.

so by default all pull_request events will have github.repository == 'scikit-hep/pyhf'. :/

---

So the question

"How do you trigger events that should be run on when a pull request from a contributor is opened/pushed to?

is still open.