Reboot when executing g-code

[AlexMoiseevs]

Controller Board

ESP32+OLED+SD the presence or absence of OLED does not affect the problem.

Machine Description

The drivers are not connected.

Input Circuits

No response

Configuration file

name: "ESP32 Dev Controller V4"
board: "ESP32 Dev Controller V4"

stepping:
  engine: RMT
  idle_ms: 255
  dir_delay_us: 1
  pulse_us: 2
  disable_delay_us: 0

axes:
  shared_stepper_disable_pin: gpio.13:high
  
  x:
    steps_per_mm: 17.777
    max_rate_mm_per_min: 100000
    acceleration_mm_per_sec2: 5000
    max_travel_mm: 1000
    homing:
      cycle: 2
      mpos_mm: 10
      positive_direction: false
    
    motor0:
      limit_all_pin: gpio.17:low:pu
      stepstick:
        direction_pin: gpio.14
        step_pin: gpio.12
    motor1:
      null_motor:

  y:
    steps_per_mm: 17.777
    max_rate_mm_per_min: 100000
    acceleration_mm_per_sec2: 5000
    max_travel_mm: 1000
    homing:
      cycle: 2
      mpos_mm: 10
      positive_direction: false

    motor0:
      limit_all_pin: gpio.4:low:pu
      stepstick:
        direction_pin: gpio.15
        step_pin: gpio.26
    motor1:
      null_motor:

  z:
    steps_per_mm: 800
    max_rate_mm_per_min: 2000
    acceleration_mm_per_sec2: 25
    max_travel_mm: 1000
    homing:
      cycle: 1
      mpos_mm: 10
      positive_direction: true

    motor0:
      limit_all_pin: gpio.16:low:pu
      stepstick:
        direction_pin: gpio.33
        step_pin: gpio.27
    motor1:
      null_motor:

spi:
  miso_pin: gpio.19
  mosi_pin: gpio.23
  sck_pin: gpio.18

sdcard:
  cs_pin: gpio.5
  card_detect_pin: NO_PIN

Startup Messages

FluidNC v1.2.0 using COM4
Exit: Ctrl-C, Ctrl-Q or Ctrl-], Clear screen: CTRL-W
Upload: Ctrl-U, Reset ESP32: Ctrl-R, Send Override: Ctrl-O
$ss
[MSG:INFO: FluidNC v3.0.x (noGit)]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.1-1-gb8050b365e]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:INFO: Configuration file:config.yaml]
[MSG:INFO: Machine ESP32 Dev Controller V4]
[MSG:INFO: Board ESP32 Dev Controller V4]
[MSG:INFO: SPI SCK:gpio.18 MOSI:gpio.23 MISO:gpio.19]
[MSG:INFO: SD Card cs_pin:gpio.5 detect:NO_PIN]
[MSG:INFO: Stepping:RMT Pulse:2us Dsbl Delay:0us Dir Delay:1us Idle Delay:255ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Shared stepper disable gpio.13]
[MSG:INFO: Axis X (10.000,1010.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:gpio.12 Dir:gpio.14 Disable:NO_PIN]
[MSG:INFO:  X All Limit gpio.17:low:pu]
[MSG:INFO:   Motor1]
[MSG:INFO: Axis Y (10.000,1010.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:gpio.26 Dir:gpio.15 Disable:NO_PIN]
[MSG:INFO:  Y All Limit gpio.4:low:pu]
[MSG:INFO:   Motor1]
[MSG:INFO: Axis Z (-990.000,10.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:gpio.27 Dir:gpio.33 Disable:NO_PIN]
[MSG:INFO:  Z All Limit gpio.16:low:pu]
[MSG:INFO:   Motor1]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Using spindle NoSpindle]
[MSG:INFO: STA SSID is not set]
[MSG:INFO: AP SSID FluidNC IP 192.168.0.1 mask 255.255.255.0 channel 1]
[MSG:INFO: AP started]
[MSG:INFO: WiFi on]
[MSG:INFO: Captive Portal Started]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]
ok

User Interface Software

No response

What happened?

When the browser is connected to esp 32, when executing a large G code, everything ends OK. If you exit the browser, then when you finish reading from the SD card (the strip ends), the system goes into reboot. Occurs on releases after 3.4.4.

Other Information

Another observation: when starting from the phone, after disabling Wi-Fi, the execution stops. Then crash.

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371
ets Jun 8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:12812
load:0x40080400,len:3032
entry 0x400805e4

[MSG:INFO Init OLED SDA:gpio.21 SCL:gpio.22]
[MSG:INFO: FluidNC v3.0.x (noGit)]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.1-1-gb8050b365e]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:INFO: Configuration file:config.yaml]
[MSG:INFO: Machine ESP32 Dev Controller V4]
[MSG:INFO: Board ESP32 Dev Controller V4]
[MSG:INFO: SPI SCK:gpio.18 MOSI:gpio.23 MISO:gpio.19]
[MSG:INFO: SD Card cs_pin:gpio.5 detect:NO_PIN]
[MSG:INFO: Stepping:RMT Pulse:2us Dsbl Delay:0us Dir Delay:1us Idle Delay:255ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Shared stepper disable gpio.13]
[MSG:INFO: Axis X (10.000,1010.000)]
[MSG:INFO: Motor0]
[MSG:INFO: stepstick Step:gpio.12 Dir:gpio.14 Disable:NO_PIN]
[MSG:INFO: X All Limit gpio.17:low:pu]
[MSG:INFO: Motor1]
[MSG:INFO: Axis Y (10.000,1010.000)]
[MSG:INFO: Motor0]
[MSG:INFO: stepstick Step:gpio.26 Dir:gpio.15 Disable:NO_PIN]
[MSG:INFO: Y All Limit gpio.4:low:pu]
[MSG:INFO: Motor1]
[MSG:INFO: Axis Z (-990.000,10.000)]
[MSG:INFO: Motor0]
[MSG:INFO: stepstick Step:gpio.27 Dir:gpio.33 Disable:NO_PIN]
[MSG:INFO: Z All Limit gpio.16:low:pu]
[MSG:INFO: Motor1]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Using spindle NoSpindle]
[MSG:INFO: STA SSID is not set]
[MSG:INFO: AP SSID FluidNC IP 192.168.0.1 mask 255.255.255.0 channel 1]
[MSG:INFO: AP started]
[MSG:INFO: WiFi on]
[MSG:INFO: Captive Portal Started]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]

Grbl 3.0 [FluidNC v3.0.x (noGit) (wifi) '$' for help]
[MSG:INFO: '$H'|'$X' to unlock]
[MSG:INFO: Channel auto report interval set to 100 ms]
[MSG:INFO: Channel auto report interval set to 100 ms]
[MSG:INFO: Caution: Unlocked]

abort() was called at PC 0x401a224f on core 1

Backtrace:0x40085e25:0x3ffb22d00x4008f3b5:0x3ffb22f0 0x4009534d:0x3ffb2310 0x401a224f:0x3ffb2390 0x401a2296:0x3ffb23b0 0x401a205b:0x3ffb23d0 0x401b997d:0x3ffb23f0 0x4010c3ff:0x3ffb2410 0x4010e961:0x3ffb2430 0x400ed211:0x3ffb2450 0x400d561d:0x3ffb24b0 0x400d5766:0x3ffb24d0 0x400edada:0x3ffb2500 0x400edb79:0x3ffb2520 0x400e1d68:0x3ffb2540 0x400dc0b9:0x3ffb2560 0x400dc70e:0x3ffb2580 0x400e1dc1:0x3ffb25a0 0x400dafd2:0x3ffb25c0 0x400eb818:0x3ffb2680 0x400ec71a:0x3ffb26a0 0x400e1bed:0x3ffb27e0 0x40110b45:0x3ffb2820

ELF file SHA256: 0000000000000000

Rebooting...
ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:12812
load:0x40080400,len:3032
entry 0x400805e4

[MSG:INFO Init OLED SDA:gpio.21 SCL:gpio.22]
[MSG:INFO: FluidNC v3.0.x (noGit)]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.1-1-gb8050b365e]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:ERR: Skipping configuration file due to panic]
[MSG:INFO: Using default configuration]
[MSG:INFO: Axes: using defaults]
[MSG:INFO: Machine Default (Test Drive)]
[MSG:INFO: Board None]
[MSG:INFO: SPI not defined]
[MSG:INFO: No SD Card CS Pin]
[MSG:INFO: See http://wiki.fluidnc.com/en/config/sd_card#sdfallbackcs-access-sd-without-a-config-file]
[MSG:INFO: Stepping:RMT Pulse:4us Dsbl Delay:0us Dir Delay:0us Idle Delay:255ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Axis X (-1000.000,0.000)]
[MSG:INFO: Axis Y (-1000.000,0.000)]
[MSG:INFO: Axis Z (-1000.000,0.000)]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Using spindle NoSpindle]
[MSG:INFO: STA SSID is not set]
[MSG:INFO: AP SSID FluidNC IP 192.168.0.1 mask 255.255.255.0 channel 1]
[MSG:INFO: AP started]
[MSG:INFO: WiFi on]
[MSG:INFO: Captive Portal Started]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]

Grbl 3.0 [FluidNC v3.0.x (noGit) (wifi) '$' for help]
[MSG:INFO: '$H'|'$X' to unlock]

[MitchBradley]

I think I know why this is happening. I will try to fix it soon. The problem has to do with websocket connections disappearing.

[AlexMoiseevs]

Found the points of occurrence of exceptions. Release FluidNC-3.6.5-pre2
Initially, the crash happens when a message is output in a file Protocol.cpp in line 149
*chan << "[MSG:" << infile->path() << " file job succeeded]\n";
When deleting it further in the file Report.cpp in line 140
channel.ack(status_code);
I deleted these drains – everything worked. I haven't noticed any other bugs yet.
I had a similar situation with this crystal on another project. When executing a large amount of code in an interrupt, a kernel timeout exception occurred. It may also be due to stack overflow.

[AlexMoiseevs]

Well, catch up. If you exit the web interface and then go through a pause and end the program – also crash. Satisfy Protocol.cpp in line 641
infile->getChannel() << "[MSG:"
<< "Reset during file job at line: " << infile->getLineNumber();

[MitchBradley]

Those are consistent with the premature closing of websockets.

[AlexMoiseevs]

There is one problem left: when reconnecting during execution, the G code
periodically crashes. I haven't found a pattern yet.
And in the 3.6.5pre 2 release, files are not uploaded to SD.

[MitchBradley]

The file upload problem is in the WebUI code. I fixed it a few days ago but have not put it in a release.

Reconnecting is a difficult problem to solve. The problem is that, when you reconnect, FluidNC has to send index.html.gz to the browser over HTTP. index.html.gz is stored in the FLASH filesystem so sending it requires a lot of FLASH accesses. FLASH accesses are very slow compared to RAM accesses. Furthermore, the FluidNC core code is also fundamentally stored in FLASH; there is not nearly enough RAM on an ESP32 - and especially not enough instruction RAM (IRAM) - for all of the FluidNC code. The ESP32 maintains a high instruction execution rate even when fetching code from FLASH by virtue of a cache. Unfortunately, cache misses do not work right when the ESP32 is inside an interrupt handler and there is an ongoing FLASH access for data, as happens when you are trying to send index.html.gz from the FLASH filesystem out to the brower and an interrupt comes along - and interrupts happen all the time during GCode execution.

It is possible to mark interrupt routines so they are permanently cached in the limited amount of IRAM. Unfortunately, there is a deficiency in the xtensa C++ compiler whereby vtables for virtual methods, and also jump tables for switch statements, are left in FLASH instead of caching them in RAM. So if you call a virtual method or code a switch statement in an ISR handler, that can cause a crash if you happen to get a cache miss on the vtable or jump table when the FLASH controller is busy accessing data from index.html.gz.

One mitigation that we use is a custom linker script that forces certain vtables into RAM. It is really hard to maintain because there is no easy way to find all the modules that have to be special-cased. And it totally broke when we changed to the new compiler and framework versions due to some naming changes in the linker scripts.

Another mitigation is that I made it so attempts to reload WebUI while running just redirect you to a page that says "sorry, can't reload right now". But I sometimes turn off that so I can do a full reload in hopes of finding vtables that need to be moved.

Even with the redirect there is a small probability of crashing because the redirect page has to be accessed from the FLASH - but it is very small so the crash window is smaller.

ESP32 achieves its extraordinary cost/performance via a collection of tricks, and when you try to make a complex realtime system with it, it is very difficult to avoid falling through one of the cracks between those tricks.

Another mitigation might be to have the option of fetching index.html.gz from SD - but then you run into the possibility of GCode timing glitches when trying to access index.html.gz and the GCode "simultaneously" from the one SD. It might work, but I can imagine a whole new layer of subtle problems that make me tear even more of my hair out - without increasing the miniscule donation rate that does not even pay for the coffee I drink while working on FluidNC.

Another possibility would be to fetch index.html.gz from the host computer. That has its own set of problems.

[MitchBradley]

Oh, ouch. It is a multiprocessor synchronization issue. When the websocket from the browser closes, the websocket disconnect event fires, which removes the websocket object from the list of active IO channels and then deletes the object. Meanwhile the other CPU core is polling all the of the active IO channels to see if any of them want to send status, and the code on that other core has already picked up the websocket object pointer from the list, so it calls a deleted object. I have to think about the best way to fix this.

[AlexMoiseevs]

Thank you for the comprehensive answer!

Offtop, I've been looking for two days, I can't find it in any way: how to disable stepper motors from the G code at the end of the program?

[bdring]

Motors are disabled with $MD.

http://wiki.fluidnc.com/en/features/commands_and_settings

[AlexMoiseevs]

Well, you underestimate me a lot...

G-code

G90
G1 X720 F100000
G1 X0

G90
G1 X720 F100000
G1 X0

$MD

Commands

$SD/Run=/test2.g
<Idle|MPos:280.981,0.000,0.000,0.000|FS:0,0|Pn:A>
<Idle|MPos:280.981,0.000,0.000,0.000|FS:0,0|Pn:A|SD:0.00,/sd/test2.g>
[MSG:INFO: Disabling all motors] <--------- does not turn off!!!!!!!!!!!!
[MSG:/sd/test2.g file job succeeded]
<Idle|MPos:280.981,0.000,0.000,0.000|FS:0,0>
$LocalFS/Run=/macro1.g
<Idle|MPos:280.981,0.000,0.000,0.000|FS:0,0|Pn:A|WCO:281.000,0.000,0.000,0.000|SD:0.00,/spiffs/macro1.g>
[MSG:INFO: Disabling all motors] <-------- manually turns off !!!!!!!
[MSG:/spiffs/macro1.g file job succeeded]
<Idle|MPos:280.981,0.000,0.000,0.000|FS:0,0>

[AlexMoiseevs]

figured it out, need a pause: G04 P1

p.s.
and there 's nothing good about it . it looks like a crutch...

[MitchBradley]

The reason the pause is needed is because $MD is not a GCode command - the NIST-standard GCode language knows nothing at all about motors. Since MD is not processed by the GCode interpreter, it cannot be automatically synchronized to the GCode execution sequence. You need to explicitly synchronize the GCode execution with a pause.

When GCode lines are processed, one line does not finish before the next one is fetched, parsed, and passed on to the motion planner. The read-ahead is necessary to ensure smooth motion across changes in speed and direction. Otherwise, the machine would have to accelerate and decelerate to a stop between every motion segment.