Internal Error when formatting a file with comments containing fmt: skip

[CodeDude56]

Describe the bug
Internal Error when reformatting file with comments at the end, where the first line has # fmt: skip added to it. All additional comment lines are removed.

To Reproduce

For example, take this code:

def foo():
    pass


# comment1  # fmt: skip
# comment2

And run it with these arguments:

$ black file.py

The resulting error is:

cannot format file.py: INTERNAL ERROR: Black 25.1.1.dev21+gdbb14ea on Python (CPython) 3.12.8 produced different code on the second pass of the formatter. Please report a bug on https://github.com/psf/black/issues. This diff might be helpful: /tmp/blk_5w5nafph.log

Mode(target_versions=set(), line_length=88, string_normalization=True, is_pyi=False, is_ipynb=False, skip_source_first_line=False, magic_trailing_comma=True, python_cell_magics=set(), preview=False, unstable=False, enabled_features=set())
--- source
+++ first pass
@@ -1,6 +1,3 @@
-def foo():
-    pass
-
-
-# comment1  # fmt: skip
-# comment2
\ No newline at end of file
+def foo():
+    pass
+# comment1  # fmt: skip
--- first pass
+++ second pass
@@ -1,3 +1,5 @@
 def foo():
     pass
+
+
 # comment1  # fmt: skip

Expected behavior

It should leave the lines of comments untouched.

Environment

My environment:

• Black's version: 25.1.0
• OS and Python version: Windows / Python (CPython) 3.10.6

This also happens on the Black Web Playground:

• Black's version: 25.1.1.dev21+gdbb14ea
• OS and Python version: Black Web Playground / Python (CPython) 3.12.8

Additional context
Originally, this was done with commented-out code (that's why # fmt: skip was in a comment), but it happens with any comment. The only thing that seems to matter is if the skip tag is in the first comment line of a block, and the block of comment lines is at the end of the file. If the skip tag is on the second line of comments, or if there is additional python code after the comment block, the format succeeds and works as expected.

[CodeDude56]

In case it's useful, running this through black with the line-ranges 5-6 argument results in a "success" even though the file is incorrectly modified to be significantly different from the original.

Does black skip the check of the original and new file when this argument is used? If so, this might be related to this vscode-black-formatter bug where commented-out code at the end of the file is being deleted when the file is saved.

[tusharsadhwani]

Yes, it is the same bug. You can verify by taking your code and running black with --fast locally:

def foo():
    pass


# comment1  # fmt: skip
# comment2

With black --fast x.py the last line disappears:

def foo():
    pass


# comment1  # fmt: skip

[tusharsadhwani]

I've narrowed down the bug a fair amount, it is within normalize_fmt_off

That function is modifying the ENDMARKER node's prefix somehow.

Before normalize_fmt_off is called, ENDMARKER has these prefixes:

[Leaf(STANDALONE_COMMENT, '# comment 1  # fmt: skip'), Leaf(STANDALONE_COMMENT, '# comment 2')]

After that function is called:

[Leaf(STANDALONE_COMMENT, '# comment 1  # fmt: skip')]

Edit: narrowed it down to _generate_ignored_nodes_from_fmt_skip being buggy.