bug: tests/test_loader.py::test_load_builtin_modules fails on PyPy3 (assumes internal CPython _json module) #362
Description
Description of the bug
The tests/test_loader.py::test_load_builtin_modules test is failing on PyPy3, as it attempts to import _json module that is specific to CPython. PyPy has only the public json module.
To Reproduce
uv venv -p pypy3.10
uv run pytest
Full traceback
Full traceback
========================================================= test session starts =========================================================
platform linux -- Python 3.10.16[pypy-7.3.18-final], pytest-8.3.4, pluggy-1.5.0
Using --randomly-seed=2897836564
rootdir: /tmp/griffe
configfile: pyproject.toml
plugins: cov-6.0.0, randomly-3.16.0, xdist-3.6.1
collected 831 items
tests/test_public_api.py .. [ 0%]
tests/test_inspector.py ............ [ 1%]
tests/test_diff.py ............................ [ 5%]
tests/test_inheritance.py ................. [ 7%]
tests/test_encoders.py .. [ 7%]
tests/test_stdlib.py .......................................................................................................... [ 20%]
...................................................................................................................... [ 34%]
tests/test_docstrings/test_numpy.py ..................................................................... [ 42%]
tests/test_expressions.py ................................................................. [ 50%]
tests/test_extensions.py ............. [ 51%]
tests/test_functions.py .......... [ 53%]
tests/test_docstrings/test_warnings.py . [ 53%]
tests/test_docstrings/test_google.py .................................................................................. [ 63%]
tests/test_nodes.py .................................................................................................... [ 75%]
tests/test_merger.py .. [ 75%]
tests/test_finder.py ................................. [ 79%]
tests/test_models.py ............................ [ 82%]
tests/test_loader.py .....................F....... [ 86%]
tests/test_visitor.py ................................... [ 90%]
tests/test_mixins.py . [ 90%]
tests/test_cli.py .... [ 91%]
tests/test_internals.py s.....s [ 91%]
tests/test_docstrings/test_sphinx.py ................................................................ [ 99%]
tests/test_git.py ... [100%]
============================================================== FAILURES ===============================================================
______________________________________________________ test_load_builtin_modules ______________________________________________________
self = <_griffe.loader.GriffeLoader object at 0x00005612c21ba678>, objspec = '_json'
def load(
self,
objspec: str | Path | None = None,
/,
*,
submodules: bool = True,
try_relative_path: bool = True,
find_stubs_package: bool = False,
) -> Object | Alias:
"""Load an object as a Griffe object, given its Python or file path.
Note that this will load the whole object's package,
and return only the specified object.
The rest of the package can be accessed from the returned object
with regular methods and properties (`parent`, `members`, etc.).
Examples:
>>> loader.load("griffe.Module")
Alias("Module", "_griffe.models.Module")
Parameters:
objspec: The Python path of an object, or file path to a module.
submodules: Whether to recurse on the submodules.
This parameter only makes sense when loading a package (top-level module).
try_relative_path: Whether to try finding the module as a relative path.
find_stubs_package: Whether to search for stubs-only package.
If both the package and its stubs are found, they'll be merged together.
If only the stubs are found, they'll be used as the package itself.
Raises:
LoadingError: When loading a module failed for various reasons.
ModuleNotFoundError: When a module was not found and inspection is disallowed.
Returns:
A Griffe object.
"""
obj_path: str
package = None
top_module = None
# We always start by searching paths on the disk,
# even if inspection is forced.
logger.debug("Searching path(s) for %s", objspec)
try:
> obj_path, package = self.finder.find_spec(
objspec, # type: ignore[arg-type]
try_relative_path=try_relative_path,
find_stubs_package=find_stubs_package,
)
src/_griffe/loader.py:140:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/_griffe/finder.py:202: in find_spec
return module_name, self.find_package(top_module_name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <_griffe.finder.ModuleFinder object at 0x00005612c21ba758>, module_name = '_json'
def find_package(self, module_name: str) -> Package | NamespacePackage:
"""Find a package or namespace package.
Parameters:
module_name: The module name.
Raises:
ModuleNotFoundError: When the module cannot be found.
Returns:
A package or namespace package wrapper.
"""
filepaths = [
Path(module_name),
# TODO: Handle .py[cod] and .so files?
# This would be needed for package that are composed
# solely of a file with such an extension.
Path(f"{module_name}.py"),
]
real_module_name = module_name
real_module_name = real_module_name.removesuffix("-stubs")
namespace_dirs = []
for path in self.search_paths:
path_contents = self._contents(path)
if path_contents:
for choice in filepaths:
abs_path = path / choice
if abs_path in path_contents:
if abs_path.suffix:
stubs = abs_path.with_suffix(".pyi")
return Package(real_module_name, abs_path, stubs if stubs.exists() else None)
init_module = abs_path / "__init__.py"
if init_module.exists() and not _is_pkg_style_namespace(init_module):
stubs = init_module.with_suffix(".pyi")
return Package(real_module_name, init_module, stubs if stubs.exists() else None)
init_module = abs_path / "__init__.pyi"
if init_module.exists():
# Stubs package
return Package(real_module_name, init_module, None)
namespace_dirs.append(abs_path)
if namespace_dirs:
return NamespacePackage(module_name, namespace_dirs)
> raise ModuleNotFoundError(module_name)
E ModuleNotFoundError: _json
src/_griffe/finder.py:274: ModuleNotFoundError
During handling of the above exception, another exception occurred:
def test_load_builtin_modules() -> None:
"""Assert builtin/compiled modules can be loaded."""
loader = GriffeLoader()
loader.load("_ast")
loader.load("_collections")
> loader.load("_json")
tests/test_loader.py:357:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/_griffe/loader.py:156: in load
top_module_object = dynamic_import(top_module_name, self.finder.search_paths)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
import_path = '_json'
import_paths = [PosixPath('/tmp/griffe'), PosixPath('/tmp/griffe/.venv/bin'), PosixPath('/usr/lib/pypy3.10'), PosixPath('/tmp/griffe/.venv/lib/pypy3.10/site-packages'), PosixPath('/tmp/griffe/src')]
def dynamic_import(import_path: str, import_paths: Sequence[str | Path] | None = None) -> Any:
"""Dynamically import the specified object.
It can be a module, class, method, function, attribute,
nested arbitrarily.
It works like this:
- for a given object path `a.b.x.y`
- it tries to import `a.b.x.y` as a module (with `importlib.import_module`)
- if it fails, it tries again with `a.b.x`, storing `y`
- then `a.b`, storing `x.y`
- then `a`, storing `b.x.y`
- if nothing worked, it raises an error
- if one of the iteration worked, it moves on, and...
- it tries to get the remaining (stored) parts with `getattr`
- for example it gets `b` from `a`, then `x` from `b`, etc.
- if a single attribute access fails, it raises an error
- if everything worked, it returns the last obtained attribute
Since the function potentially tries multiple things before succeeding,
all errors happening along the way are recorded, and re-emitted with
an `ImportError` when it fails, to let users know what was tried.
IMPORTANT: The paths given through the `import_paths` parameter are used
to temporarily patch `sys.path`: this function is therefore not thread-safe.
IMPORTANT: The paths given as `import_paths` must be *correct*.
The contents of `sys.path` must be consistent to what a user of the imported code
would expect. Given a set of paths, if the import fails for a user, it will fail here too,
with potentially unintuitive errors. If we wanted to make this function more robust,
we could add a loop to "roll the window" of given paths, shifting them to the left
(for example: `("/a/a", "/a/b", "/a/c/")`, then `("/a/b", "/a/c", "/a/a/")`,
then `("/a/c", "/a/a", "/a/b/")`), to make sure each entry is given highest priority
at least once, maintaining relative order, but we deem this unnecessary for now.
Parameters:
import_path: The path of the object to import.
import_paths: The (sys) paths to import the object from.
Raises:
ModuleNotFoundError: When the object's module could not be found.
ImportError: When there was an import error or when couldn't get the attribute.
Returns:
The imported object.
"""
module_parts: list[str] = import_path.split(".")
object_parts: list[str] = []
errors = []
with sys_path(*(import_paths or ())):
while module_parts:
module_path = ".".join(module_parts)
try:
module = import_module(module_path)
except BaseException as error: # noqa: BLE001
# pyo3's PanicException can only be caught with BaseException.
# We do want to catch base exceptions anyway (exit, interrupt, etc.).
errors.append(_error_details(error, module_path))
object_parts.insert(0, module_parts.pop(-1))
else:
break
else:
> raise ImportError("; ".join(errors))
E ImportError: With sys.path = ['/tmp/griffe', '/tmp/griffe/.venv/bin', '/usr/lib/pypy3.10', '/tmp/griffe/.venv/lib/pypy3.10/site-packages', '/tmp/griffe/src'], accessing '_json' raises ModuleNotFoundError: No module named '_json'
src/_griffe/importer.py:107: ImportError
========================================================== warnings summary ===========================================================
tests/test_stdlib.py::test_fuzzing_on_stdlib[ctypes]
/usr/lib/pypy3.10/ctypes/test/test_structures.py:335: DeprecationWarning: invalid escape sequence ')'
"\(Phone\) .*TypeError.*: expected bytes, int found")
tests/test_stdlib.py::test_fuzzing_on_stdlib[ctypes]
/usr/lib/pypy3.10/ctypes/test/test_structures.py:340: DeprecationWarning: invalid escape sequence ')'
"\(Phone\) .*TypeError.*: too many initializers")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
FAILED tests/test_loader.py::test_load_builtin_modules - ImportError: With sys.path = ['/tmp/griffe', '/tmp/griffe/.venv/bin', '/usr/lib/pypy3.10', '/tmp/griffe/.venv/lib/pypy3.10/site-pa...
======================================== 1 failed, 828 passed, 2 skipped, 2 warnings in 14.02s ========================================Expected behavior
Tests passing.
Environment information
- System: Linux-6.13.4-gentoo-dist-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-glibc2.41
- Python: pypy 7.3.18 (/tmp/griffe/.venv/bin/python3)
- Environment variables:
- Installed packages:
griffev1.5.8.dev1+g9717739
Additional context
I can submit a pull request to fix this. Just please let me know if you prefer that I guarded _json test with a PyPy (or CPython) check, or just removed test for that module entirely (and possibly used another module that's common to CPython and PyPy).