[BUG] Random freezes during printing over serial connection (skr 1.4 turbo)

[m-kozlowski]

Bug Description

I'm experiencing radom freezes when printing with Octoprint over serial connection.
Using SKR 1.4 turbo, RPI4, directly connected to TFT serial interface (SERIAL_PORT 0).
Running bugfix-2x 004bed8
I am not able to identify any specific circumstances of the error - sometimes it happens on first layer, and sometimes after few hours of printing the same gcode.
Print head stops, screen (ulti controler) stops refreshing, communication freezes (there is "timeout" message in octoprint terminal window).

I'm suspecting it's related to recent serial changes, because:

• I'm not experiencing this issue when printing from SD card
• I've reverted to 811bb79 and so far it seems to be working fine.

Configuration Files

Configuration.zip

[rhapsodyv]

Can you test using another cable and another computer?

Skr uses usb serial, that is a lot stable than normal hardware serial. In the past, some users reported something similar and the issue was in the host.

[m-kozlowski]

Sure, will try. But it will be another rpi4 as I don't have anything else with hardware serial at hand.

However I'm not convinced if it's host to blame. It has printed hundreds of hours in this setup. And right now it's finishing ~18 hour print running downgraded Marlin.

[rhapsodyv]

@X-Ryl669 can it be related with #20783 and #20932 ?

[rhapsodyv]

Which version of marlin is working fine for you?

[X-Ryl669]

What happen when the serial is freezing ? Do you have emergency parser enabled (previously) ? If yes, can you send some emergency commands to see if it react ?

Can you post the last serial output log so we can see what's the last message that worked ?

[X-Ryl669]

I've read your configuration file.

I'm trying to validate some hypothesis here:

1. Checking if the printer is crashed when it happens
2. Checking if the g-code parser is waiting for more data and is struck
3. Checking if my changes have changed the behavior of the printer

For 1:

• Please pull Add post mortem debugging #20492 PR and build it (you'll need to enable MARLIN_DEV_MODE and POSTMORTEM_DEBUGGING in Configuration_adv.h) then when it crashes, copy the output from the serial output here

For 2:

• Enable emergency parser in your Configuration_adv.h
• When it happens, try to send a M108 manually and see if it's answering

For 3:

• If none of the previous test are successful, please say so, I'll try to contact you directly to send you modified version of sources file to spot what happens.

[m-kozlowski]

Applied changes as described above and started ~4 hours print with serial communication logging enabled.
Will report back.

In the meantime: I ran few short prints overnight, and was unable to trigger issue. Both on new-rpi4 and old-rpi4.
The only thing that changed is that i plugged USB cable in, while still printing on hardware serial.
Not much to base any conclusions, but it might be some grounding issue (?). Gonna investigate further.

[m-kozlowski]

Today I haven't been able to repeat the problem on any of the two raspberry pi's and any version of Marlin.
I'll leave MARLIN_DEV_MODE and EMERGENCY_PARSER enabled and reopen the ticket if I capture anything.
Meanwhile, I am closing the ticket, recognizing that it could be a wiring problem.

[X-Ryl669]

Or it's a bug that's disappearing when EMERGENCY_PARSER is enabled... Feel free to reopen if it re-appear once you disable emergency parser.

[m-kozlowski]

Happened again.
Bad news is I was running 9d24ee8 without "post mortem" PR.
Better news is that EMERGENCY_PARSER and MARLIN_DEV_MODE was on.

Printer does not react to M108.
Last lines of serial communication:

Send: N119024 M117 98% L=225/240*98
Recv: //action:notification 98% L=225/240
Recv: ok N119024 P0 B3
Send: N119025 G1 F3840.000*97
Recv: ok N119025 P1 B3
Send: N119026 G1 X109.294 Y110.515 E-0.19172*120
Recv: ok N119026 P0 B3
Send: N119027 G1 X109.518 Y110.953 E-0.18245*114
Recv: ok N119027 P0 B3
Send: N119028 G1 X109.663 Y111.429 E-0.18478*123
Recv: ok N119028 P0 B3
Send: N119029 G1 X109.706 Y111.711 E-0.10580*126
Recv: ok N119029 P0 B3
Send: N119030 G1 X109.722 Y112.000 E-0.10737*122
Recv: ok N119030 P0 B3
Send: N119031 G1 X109.706 Y112.289 E-0.10737*126
Recv: ok N119031 P0 B3
Send: N119032 G1 X109.663 Y112.571 E-0.10580*113
Recv: ok N119032 P0 B3
Send: N119033 G1 X109.518 Y113.047 E-0.18478*112
Recv: ok N119033 P0 B3
Send: N119034 G1 X109.294 Y113.485 E-0.18245*118
Recv: ok N119034 P0 B3
Send: N119035 G1 X109.129 Y113.721 E-0.10706*117
Recv: ok N119035 P0 B3
Send: N119036 G1 X108.938 Y113.941 E-0.10789*112
Recv: ok N119036 P0 B3
Send: N119037 G1 X108.727 Y114.138 E-0.10724*119
Recv: ok N119037 P0 B3
Send: N119038 G1 X108.489 Y114.318 E-0.11077*127
Recv: ok N119038 P0 B3
Send: N119039 G1 X108.248 Y114.463 E-0.10426*127
Recv: ok N119039 P0 B3
Send: N119040 G1 X107.790 Y114.659 E-0.18490*112
Recv: ok N119040 P0 B3
Send: N119041 G1 X107.507 Y114.735 E-0.10892*112
Recv: ok N119041 P0 B3
Send: N119042 G1 X107.218 Y114.781 E-0.10849*115
Recv: ok N119042 P0 B3
Send: N119043 G1 X106.925 Y114.796 E-0.10868*115
Recv: ok N119043 P0 B3
Send: N119044 G1 X106.633 Y114.781 E-0.10868*122
Recv: ok N119044 P0 B3
Send: N119045 G1 X106.348 Y114.736 E-0.10704*123
Recv: ok N119045 P0 B3
Send: N119046 G1 X105.875 Y114.591 E-0.18365*122
Recv: ok N119046 P0 B3
Send: N119047 G1 X105.613 Y114.469 E-0.10731*126
Recv: ok N119047 P0 B3
Send: N119048 G1 X105.365 Y114.320 E-0.10717*123
Recv: ok N119048 P0 B3
Send: N119049 G1 X104.980 Y114.008 E-0.18388*121
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N119050 M105*27
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N119051 M105*26
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N119052 M105*25
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N119053 M105*24
Communication timeout while printing, trying to trigger response from printer. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Send: N119054 M105*31
No response from printer after 6 consecutive communication timeouts, considering it dead. Configure long running commands or increase communication timeout if that happens regularly on specific commands or long moves.
Changing monitoring state from "Printing" to "Offline (Error: Too many consecutive timeouts, printer still connected and alive?)"
Connection closed, closing down monitor

I have st-link v2 plugged in that I'm using to flash firmware, but I guess it can be used to do some debugging if someone guide me. I can leave the printer in current state for a while.

[X-Ryl669]

Ok, so I hope you've openocd installed. In that case, you'll have to start the server:

$ ~/.platformio/packages/tool-openocd/bin/openocd -d2 -s ~/.platformio/packages/tool-openocd/scripts -f interface/stlink.cfg  -f target/stm32f1x.cfg

Replace stm32f1x.cfg by the configuration for your CPU

Then in another terminal, run gdb like this:

$ ~/.platformio/packages/toolchain-gccarmnoneeabi/bin/arm-none-eabi-gdb /path/to/your/marlin/repository/.pio/build/your_env/debug/firmware.elf

While in GDB, type:

(gdb) tar ext:3333
Remote debugging using :3333
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0801b272 in ?? ()
(gdb) bt

And post the output of the commands above

If you are a bit lost, I've written a post about how to debug Marlin and there for setting up OpenOCD.
Skip the part about unlocking the flash since you don't have a STM32.