weakrefs to dead objects occuring when changing backends in pyhf benchmark #2311
Description
Summary
I try to perform a benchmark of pyhf using pytest-benchmark quite similarly to the benchmark in the tests/benchmarks directory.
A short example of such a simple benchmark is given below. To reproduce this bug, the python code needs to be saved in a file of the format test_<name>.py and executed via pytest test_<name>.py.
The bug occurs only sometimes when the backend is changed between different benchmarking cases. Since the occurence of the bug includes some amount of randomness, it may happen that it doesn't occur on the first try but that the benchmark must be executed multiple times. The full benchmark takes around 1 min on my machine.
The suspected origin of this bug is that every time the backend is changed, an event called tensorlib_changed is triggered that in turn leads to the execution of some _precompute() functions on different objects (e.g. a TensorViewer object as in the error message). The problem occurs, when the referenced object no longer exists, as all references to it have been removed. The reference used to call the function does not change this as it is a weakref.
A proposed solution can be found in PR #2310.
OS / Environment
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammySteps to Reproduce
# content of test_benchmark.py
import pytest
import pyhf
@pytest.fixture(
scope='function',
params=[
(pyhf.tensor.numpy_backend(), None),
(pyhf.tensor.pytorch_backend(), None),
(pyhf.tensor.pytorch_backend(precision='64b'), None),
(pyhf.tensor.tensorflow_backend(), None),
(pyhf.tensor.jax_backend(), None),
(
pyhf.tensor.numpy_backend(poisson_from_normal=True),
pyhf.optimize.minuit_optimizer(),
),
],
ids=['numpy', 'pytorch', 'pytorch64',
'tensorflow',
'jax', 'numpy_minuit'],
)
def backend(request):
# get the ids of all the backends
param_ids = request._fixturedef.ids
# the backend we're using: numpy, tensorflow, etc...
param_id = param_ids[request.param_index]
# name of function being called (with params), the original name is .originalname
func_name = request._pyfuncitem.name
pyhf.set_backend(*request.param)
yield request.param
def hypotest(data, pdf):
return pyhf.infer.hypotest(1.0, data, pdf, test_stat="qtilde", return_expected=True)
bins = [1, 2, 4, 8, 16, 32]
bin_ids = [f'{n_bins}_bins' for n_bins in bins]
@pytest.mark.parametrize('n_bins', bins, ids=bin_ids)
def test_hypotest(benchmark, backend, n_bins):
model = pyhf.simplemodels.uncorrelated_background(signal=[12.0]*n_bins, bkg=[50.0]*n_bins, bkg_uncertainty=[5.0]*n_bins)
data = [51.0]*n_bins + model.config.auxdata
assert benchmark(hypotest, data, model)pytest test_benchmark.pyFile Upload (optional)
No response
Expected Results
The expected behavior is to output the benchmarking results for all considered cases as it can be observed when executing pytest in pyhf/tests/benchmarks/.
This output should not show any "test failures" as no normal tests are performed but only functions that run without an error, when called outside of the benchmark.
Actual Results
_________________________ ERROR at setup of test_hypotest[jax-1_bins] _________________________
request = <SubRequest 'backend' for <Function test_hypotest[jax-1_bins]>>
@pytest.fixture(
scope='function',
params=[
(pyhf.tensor.numpy_backend(), None),
(pyhf.tensor.pytorch_backend(), None),
(pyhf.tensor.pytorch_backend(precision='64b'), None),
(pyhf.tensor.tensorflow_backend(), None),
(pyhf.tensor.jax_backend(), None),
(
pyhf.tensor.numpy_backend(poisson_from_normal=True),
pyhf.optimize.minuit_optimizer(),
),
],
ids=['numpy', 'pytorch', 'pytorch64',
'tensorflow',
'jax', 'numpy_minuit'],
)
def backend(request):
# get the ids of all the backends
param_ids = request._fixturedef.ids
# the backend we're using: numpy, tensorflow, etc...
param_id = param_ids[request.param_index]
# name of function being called (with params), the original name is .originalname
func_name = request._pyfuncitem.name
> pyhf.set_backend(*request.param)
test_hypo_pyhf.py:29:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../pyhfDev/pyhf/src/pyhf/events.py:161: in register_wrapper
result = func(*args, **kwargs)
../../pyhfDev/pyhf/src/pyhf/tensor/manager.py:193: in set_backend
events.trigger("tensorlib_changed")()
../../pyhfDev/pyhf/src/pyhf/events.py:70: in __call__
func()(arg(), *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = None
def _precompute(self):
tensorlib, _ = get_backend()
> self.sorted_indices = tensorlib.astensor(self._sorted_indices, dtype='int')
E AttributeError: 'NoneType' object has no attribute '_sorted_indices'
../../pyhfDev/pyhf/src/pyhf/tensor/common.py:33: AttributeErrorpyhf Version
pyhf, version 0.7.1.dev116Code of Conduct
- I agree to follow the Code of Conduct