mutmut run failed to collect stats

[15r10nk]

I tried to test inline-snapshot with mutmut but got the following error:

❯ hatch run mutmut run
generating mutants
    done in 2062ms
⠇ running stats
    done
failed to collect stats, no active tests found
⠸ running clean tests........................................................,..,,,,,....
............. [ 19%]
........F
======================================= FAILURES ========================================
_____________________________________ test_persist ______________________________________

project = <tests.conftest.project.<locals>.Project object at 0x7f63af037050>

    def test_persist(project):
    
        project.setup(
            """\
    from inline_snapshot import external
    
    def test_something():
        assert "hello" == snapshot(external("bbbbb*.txt"))
        assert 2 == snapshot(1+1)
    """
        )
    
        result = project.run("--inline-snapshot=update")
    
>       assert project.storage() == snapshot([])

/home/frank/projects/inline-snapshot/mutants/tests/test_external.py:76: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/frank/projects/inline-snapshot/src/inline_snapshot/_inline_snapshot.py:529: in __ca
ll__
    return self.func(*args, **kwargs)
/home/frank/projects/inline-snapshot/src/inline_snapshot/_inline_snapshot.py:593: in snap
shot
    snapshots[key]._re_eval(obj)
/home/frank/projects/inline-snapshot/src/inline_snapshot/_inline_snapshot.py:642: in _re_
eval
    self._value._re_eval(obj)
/home/frank/projects/inline-snapshot/src/inline_snapshot/_inline_snapshot.py:111: in _re_
eval
    re_eval(self._old_value, self._ast_node, value)
/home/frank/projects/inline-snapshot/src/inline_snapshot/_inline_snapshot.py:95: in re_ev
al
    old_items = adapter.items(old_value, node)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <inline_snapshot._adapter.sequence_adapter.ListAdapter object at 0x7f63a2881c40>
value = [], node = None

    def items(self, value, node):
    
>       assert isinstance(node, self.node_type), (node, self)
E       AssertionError: (None, <inline_snapshot._adapter.sequence_adapter.ListAdapter obj
ect at 0x7f63a2881c40>)

/home/frank/projects/inline-snapshot/src/inline_snapshot/_adapter/sequence_adapter.py:22:
 AssertionError
--------------------------------- Captured stdout call ----------------------------------
write code:
# äöß 🐍
from inline_snapshot import snapshot
from inline_snapshot import outsource
from inline_snapshot import external

def test_something():
    assert "hello" == snapshot(external("bbbbb*.txt"))
    assert 2 == snapshot(1+1)

running: pytest --inline-snapshot=update
     in: /tmp/pytest-of-frank/pytest-396/test_persist1
============================= test session starts ==============================
platform linux -- Python 3.12.6, pytest-8.3.3, pluggy-1.5.0
rootdir: /tmp/pytest-of-frank/pytest-396/test_persist1
plugins: inline-snapshot-0.13.3, time-machine-2.16.0, hypothesis-6.115.3, subtests-0.13.1
, xdist-3.6.1
collected 1 item

test_file.py .                                                           [100%]
=============================== inline snapshot ================================
──────────────────────────────── Fix snapshots ─────────────────────────────────
+-------------------------------- test_file.py --------------------------------+
| @@ -4,5 +4,5 @@                                                              |
|                                                                              |
|  from inline_snapshot import external                                        |
|                                                                              |
|  def test_something():                                                       |
| -    assert "hello" == snapshot(external("bbbbb*.txt"))                      |
| +    assert "hello" == snapshot("hello")                                     |
|      assert 2 == snapshot(1+1)                                               |
+------------------------------------------------------------------------------+
These changes are not applied.
Use --inline-snapshot=fix to apply them, or use the interactive mode with 
--inline-snapshot=review

─────────────────────────────── Update snapshots ───────────────────────────────
+-------------------------------- test_file.py --------------------------------+
| @@ -5,4 +5,4 @@                                                              |
|                                                                              |
|                                                                              |
|  def test_something():                                                       |
|      assert "hello" == snapshot(external("bbbbb*.txt"))                      |
| -    assert 2 == snapshot(1+1)                                               |
| +    assert 2 == snapshot(2)                                                 |
+------------------------------------------------------------------------------+
These changes will be applied, because you used --inline-snapshot=update


============================== 1 passed in 0.13s ===============================
==================================== inline snapshot ====================================
=========================== short test summary info ============================
FAILED tests/test_external.py::test_persist - AssertionError: (None, <inline_...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
1 failed, 83 passed, 6 subtests passed in 8.93s
failed to run clean test

I don't know if this problem is only caused by mutmut, because inline-snapshot does also some special things with pytest/assert-rewriting and some ast analysis which can have problems when the source code is modified before it is compiled.

I hope you can reproduce the problem in this branch with.

> hatch run mutmut run

[boxed]

failed to collect stats

Hmm. I guess mutmut should error out there already. Without that step working, running the tests clean is kinda useless.

These changes will be applied, because you used --inline-snapshot=update

Are you running tests that themselves modify tests in-place on disk?

[15r10nk]

No, I'm not changing any test files in the tests/ folder on disk.

I create tests for pytest in temporary folders and use pytest to run these tests.
The update you see in the log is from one of these pytest runs (rootdir is a temporary dir).

My assumption is that my tests fail for some reason when you try to collect your stats. But I have currently no idea why.

[15r10nk]

I added a requirements.txt to my branch and tested it in a normal venv.
I got the same problem with mutmut run, but I was able to execute python -m pytest tests --import-mode=append -x -q inside the mutants directory successfully.

My best guess at the moment is that your stats collector plugin causes some problems in combination with my tests. I will later take a look at it again.

[boxed]

Getting closer... I discovered that when I go into mutants and execute pytest it still imports from the original src dir, and not from the mutants dir.

Yea.. as I write that I get why. I don't use a src dir for my projects, so after mutation generation I would get mutants/iommi for example, while you get mutants/src/inline_snapshot, but mutants is added to the python path, while you need mutants/src to be added!

[boxed]

    src_path = (Path('mutants') / 'src')
    source_path = (Path('mutants') / 'source')
    if src_path.exists():
        sys.path.insert(0, str(src_path.absolute()))
    elif source_path.exists():
        sys.path.insert(0, str(source_path.absolute))
    else:
        sys.path.insert(0, os.path.abspath('mutants'))

instead of just inserting mutants works. Now it crashes with SyntaxError: from __future__ imports must occur at the beginning of the file. Huge progress!

[boxed]

Now I discovered that my trampolines don't preserve generator semantics. So that's where I'm at right now.

[boxed]

I released 3.1.0 with fixes for from __future__, stat hard exit, and generator handling. Still no luck in running your project though, but I'm getting a lot further.

[15r10nk]

my generators return also values.

I think you should change your trampoline code to something like this:

    if not mutant_under_test.startswith(prefix):
        result=yield from orig(*args, **kwargs)
        return result # for the yield case
    mutant_name = mutant_under_test.rpartition('.')[-1]
    result = yield from mutants[mutant_name](*args, **kwargs)
    return result

[boxed]

Ah. Good point. I pushed a fix for that. Even with that fix I crash here:

  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_inline_snapshot.py", line 2297, in _get_changes
    yield from _mutmut_yield_from_trampoline(object.__getattribute__(self, "xǁUndecidedValueǁ_get_changes__mutmut_orig"), object.__getattribute__(self, "xǁUndecidedValueǁ_get_changes__mutmut_mutants"), *args, **kwargs) 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_inline_snapshot.py", line 35, in _mutmut_yield_from_trampoline
    result = yield from orig(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_inline_snapshot.py", line 1541, in xǁUndecidedValueǁ_get_changes__mutmut_orig
    yield from handle(self._ast_node, self._old_value)
  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_inline_snapshot.py", line 1522, in handle
    for item in adapter.items(obj, node):
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_adapter/dict_adapter.py", line 294, in items
    return _mutmut_trampoline(object.__getattribute__(self, "xǁDictAdapterǁitems__mutmut_orig"), object.__getattribute__(self, "xǁDictAdapterǁitems__mutmut_mutants"), *args, **kwargs) 
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_adapter/dict_adapter.py", line 14, in _mutmut_trampoline
    result = orig(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boxed/Projects/inline-snapshot/mutants/src/inline_snapshot/_adapter/dict_adapter.py", line 60, in xǁDictAdapterǁitems__mutmut_orig
    assert isinstance(node, ast.Dict), f'{type(node)}, {ast.unparse(node.parent.parent)}'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: <class 'ast.Starred'>, result = orig(*args, **kwargs)

I don't really understand how this happens.

[15r10nk]

I asked me while reading your trampoline code if it could be possible to remove the trampoline indirection and replace it with if-else constructs in the functions, like:

def add(a,b):
   if __mutmut_use_orig(...):
       return a+b
   elif __mutmut_use_mutant(1,...):
       return a-b
   elif __mutmut_use_mutant(2,...):
       return ... # some other mutant

This would allow it to use inspect.currentframe() like I do in snapshot().
The global and class namespaces would also stay clean, which can be important for code that uses globals() or cls.__dict__

[boxed]

Hm. Yea I guess that could work too. The function bodies would be rather huge which can have other problems I guess. It would also make it more annoying to implement the code to get the diff for a mutant.

I tried adding

    while 'mutmut' not in frame.f_code.co_name:
        frame = frame.f_back

into snapshot(), which maybe improves things as far as mutmut is concerned, but still it fails on that AST traversal thing.

[15r10nk]

are you not mutating snapshot() because it uses inspect.currentframe or because it has a decorator?

[boxed]

I didn't realize until you said so right now that snapshot() isn't mutated! I don't know why. It should be!

[15r10nk]

I removed the decorator and got mutated.

I think the problem is that snapshot() has problems finding the correct frame. I will look at it later again. Maybe this helps you.

[boxed]

Yea ok, so decorators stop mutation! I wrote a small test for that and yea. That should just result in fewer mutants though, not a fail like the traceback above I think.

I think the problem is that snapshot() has problems finding the correct frame. I will look at it later again. Maybe this helps you.
Yea, the added while loop I wrote about above will work around that issue.