Support performance testing of import latency

[jaraco]

In python/cpython#114664, I was experimenting with using pytest-perf to capture the import latency of importlib_metadata but learned that timeit may not support that use-case readily.

Let's explore if there's something pytest-perf can do to make measurement of import latency straightforward.

[ofek]

Have you had time to think more about this? I am very interested!

[jaraco]

I haven't yet. If/when I do, I'll update here. If you're interested in exploring the landscape, let me know and I can give you a tour of this project and how I might go about it. Basically, it boils down to creating "exercises" (Python-syntax functions that perform the warmup and benchmark logic) that are somehow customized for checking execution time at import (probably python -m importtime as described here). Whether there should be an explicit decorator for indicating to check import time or if it should be inferred from the structure of the benchmark logic, I'm not yet sure. I could imagine the test to look something like:

def import_time_perf():
    import importlib_metadata

And pytest-perf could infer that because the only statement in the test is an import statement, it should use the "import time" check.

Or it could be explicit:

from pytest_perf.deco import import_time

@import_time('importlib_metadata')
def import_time_perf():
    import importlib_metadata

But that doesn't feel quite right, because "importlib_metadata" has to be specified twice.

Maybe instead, since currently the presence of perf in the function name is used to check performance, maybe _import_time could be used to detect import time:

def check_import_time():
    import importlib_metadata

I think I'm liking that approach best.