[BUG] Adafruit MAX31865 causes serious delays on SKR 1.4 (bugfix-2.0)

[zeleps]

Did you test the latest bugfix-2.0.x code?

Yes, and the problem still exists.

Bug Description

Installing an Adafruit MAX31865 on my SKR 1.4 (as per GadgetAngel's instructions) caused big delays in almost any function. Homing pauses for 3-4 seconds before starting, and another 4'' before deploying bltouch. Prints pause for a few seconds at random steps and ultimately fail with MAXTEMP. Changing TEMP_SENSOR_0 back to 5 (my previous setup) without physically changing any connections brings speed back to normal (temperature readings are incorrect, of course).

I put some timing code for testing and found out that max865ref.readRTD() takes some 70-90msec to complete, so does max31865_0.temperature(MAX31865_SENSOR_OHMS_0, MAX31865_CALIBRATION_OHMS_0). This seems to be excessive, and makes the sensor practically useless.

I installed the sensor on the LCD's SPI at first, using pin 0.03 for CS, then changed it to on-board SPI (0.07-0.08-0.09) with 0.26 for CS. Results were the same. I'm not using SD for printing, I print directly from OctoPrint.

Bug Timeline

Issue started immediately after installing MAX31865.

Expected behavior

Little, if any, delays.

Actual behavior

Long, noticeable and ultimately fatal delays

Steps to Reproduce

No response

Version of Marlin Firmware

latest bugfix 2.0

Printer model

Custom

Electronics

SKR 1.4

Add-ons

bltouch, Adafruit MAX31865 on T0, type 5 thermistor on T1.

Your Slicer

Cura

Host Software

OctoPrint

[zeleps]

The adafruit MAX31865 library implements readRTD() as follows:

   clearFault();
    enableBias(true);
    DELAY_US(10000);
    uint8_t t = readRegister8(MAX31856_CONFIG_REG);
    t |= MAX31856_CONFIG_1SHOT;
    writeRegister8(MAX31856_CONFIG_REG, t);
    DELAY_US(65000);

    uint16_t rtd = readRegister16(MAX31856_RTDMSB_REG);

So, there's a blocking delay of 75msec in every invocation, several times per second.

It's really impressive that nobody else has complained about this till now.

There are two possible paths: one (which I'll try first) is to create a new fork of the adafruit library that implements a non blocking tryReadRTD() method, that will use millis to advance to the next step each time it is invoked, and return a fresh value only when the necessary delays have been observed.

The other one is to implement the non-blocking MAX31865.readRTD() function in Marlin. I'm not sure about the policies regarding such an implementation, so I'd like some feedback here.

[GadgetAngel]

The adafruit MAX31865 library implements readRTD() as follows:

   clearFault();
    enableBias(true);
    DELAY_US(10000);
    uint8_t t = readRegister8(MAX31856_CONFIG_REG);
    t |= MAX31856_CONFIG_1SHOT;
    writeRegister8(MAX31856_CONFIG_REG, t);
    DELAY_US(65000);

    uint16_t rtd = readRegister16(MAX31856_RTDMSB_REG);

So, there's a blocking delay of 75msec in every invocation, several times per second.

It's really impressive that nobody else has complained about this till now.

There are two possible paths: one (which I'll try first) is to create a new fork of the adafruit library that implements a non blocking tryReadRTD() method, that will use millis to advance to the next step each time it is invoked, and return a fresh value only when the necessary delays have been observed.

The other one is to implement the non-blocking MAX31865.readRTD() function in Marlin. I'm not sure about the policies regarding such an implementation, so I'd like some feedback here.

What setup are you using? Which printer?

[zeleps]

It's a custom build, based on SKR 1.4. I used to have an e3d analog amplifier for my PT100 sensor, but I smh managed to blow that up, so I recently went for an adafruit MAX31865. Thanks to your great guide (kudos for that!) I found out all the quirks for installing the new amplifier (in fact, and since my board is crammed with sensors and most pins are in use, I've installed the adafruit on the same SPI with my reprap display, just using P0.03 for CS, with no problems). But when I tried it out, I noticed large delays in all movement (homing, calibration, printing).

Yesterday I dug into your adafruit library and found the delays I describe above. I made some changes to the library in order to remove the delays (https://github.com/zeleps/Adafruit-MAX31865-V1.1.0-Mod-M.git) and replace them with timed waits and everything works fine now.

Only thing is that I occasionally read a 0xffff from the RTDMSB register, I'll probably tackle the odd error by repeating the last read once or twice before sending a reading error and halting the printer.

I'm really wondering how these delays worked for you. Marlin reads RTD from the sensor several times per second, and the 75msec delay is substantial and clearly visible.

[descipher]

It looks like its set to one shot mode, so you have to wait for it all the time. What about always on?
Have a look at this code. https://os.mbed.com/teams/Maxim-Integrated/code/MAX31856//file/a1bbb5c254f2/MAX31856.h

[zeleps]

It looks like its set to one shot mode, so you have to wait for it all the time. What about always on?
Have a look at this code. https://os.mbed.com/teams/Maxim-Integrated/code/MAX31856//file/a1bbb5c254f2/MAX31856.h

As I understand, the bias current heats the sensor, thus making reading inaccurate, so the library probably adopted this on/off strategy to keep heating at a minimum, but I guess that the reading is so frequent that the sensor will heat up anyway. The code you sent is for MAX31856, not 865, but I'll try that suggestion (leaving the sensor on) and see what happens.

In my trials yesterday, I found out that after some actual printing time, the sensor starts returning 0xFFFF until reset (power toggle). Reading the fault register, I'm getting 0x80 (MAX31865_FAULT_HIGHTHRESH). This doesn't seem to happen when not printing. I'll make some more tests to figure out the issue.

[zeleps]

Ok, I read a bit more about MAX31865 and decided that 1-shot mode is futile in the case of hotends, since reading is too frequent to prevent the bias current heating effect, plus the fact that bias current affects room temperatures, not 200°C temperatures. So I changed my MAX31865 version of the library to work in auto mode, and to simply read the rtd register, which is efficient, quick and simple. And apparently works just as fine (@descipher thanks for the tip!)

While printing, I still get some random read errors, which would normally cause a _temp_error(). Since they are single read errors I chose to ignore them at the MAX31865 library level (under certain conditions, eg. up to 5 consecutive errors or if the temp change is unnaturally big within a small period of time).

I'll investigate the odd error issue a bit more and come back. Meanwhile, since nobody else has complained about these issues, I'm planning to keep my solution to myself. If more people are interested in them, we can continue this discussion.

[GadgetAngel]

I like your solution, but you will get temperature reading errors because if you read the data "too fast" the MAX31865 sensor needs additional time to "settle out" before you take the actual reading. Why do you think reading the temperatures faster is better? Temperature changes do not occur faster than 1 sec. Using a non-blocking wait is the correct answer!! If I remember correctly, my modified library was based on the library created by Adafruit for the MAX31865. You probably are not even aware that before I did my modified library for the MAX31865 no one using a SKR 1.3/1.4 board could not use a PT100 with a MAX31865 board . The code would not even compile or run. So one shot mode, with non-blocking wait between temperature readings, is in my humble opinion, is the correct answer. When I get back into town, I will look at your code to see how you implemented the non-blocking function and add it to my library. When I tried to read the MAX31865 temperature readings faster, I experienced the temperature reading errors as you did. Marlin has the ability to reduce temperature reading errors by using the Marlin THERMOCOUPLE_MAX_ERRORS variable in configuation_adv.h file. It is defaulted to 15. So, if Marlin experiences MAXTEMP or MINTEMP errors that are less then 15 the 3D printer is not halted. Try raising the THERMOCOUPLE_MAX_ERRORS to see if it helps reduce the detections of MAX31865 temperature reading errors. Use one shot mode. Add non-blocking waits between readings. Adjust the non-blocking wait until the temperature error readings go away. I never really test the wait time to see what the minimum would be to stop the errors in the temperature readings. But you should not need to drop bad temperature reading inside the library, because Marlin has already done this. When I get back I will add your non-blocking waits to my library. My 3D printer right now is torn apart for upgrades. When I get it all back together I will adjust my non-blocking waits when I can test it on a full blown working 3D printer. But I will say this again, temperature does not changes in the span of milliseconds it changes in "number of seconds". My mistake was using blocking waits.

…

On Mon, Apr 26, 2021 at 9:36 AM zeleps ***@***.***> wrote: Closed #21638 <#21638>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21638 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH7LC2NYW27VOZ5QHQAXG2TTKVT5FANCNFSM43DBRRMQ> .

[zeleps]

Ok, this seems to be an issue with temperature.cpp.

I'm not the one polling for temperature, it's Marlin. Check this log from my serial port (each line is an invocation to readRTD()):

2021-04-26 18:12:48,006 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,016 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,026 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,036 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,046 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,057 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,058 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,066 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,076 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,086 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,096 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,106 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,116 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,126 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,136 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,148 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,156 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,166 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,176 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,186 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,196 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,206 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,216 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,226 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,236 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,246 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,256 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,266 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,276 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,286 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,297 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,306 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,308 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,316 - Recv: RTD MSB: 105 RTD LSB: 94 Fault: 0
2021-04-26 18:12:48,326 - Recv: RTD MSB: 105 RTD LSB: 94 Fault: 0
2021-04-26 18:12:48,337 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,346 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,356 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,366 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,377 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,429 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,454 - Recv: RTD MSB: 105 RTD LSB: 94 Fault: 0
2021-04-26 18:12:48,479 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,486 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,496 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,506 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,516 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,526 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,537 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,546 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,556 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,557 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,566 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,576 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,586 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,596 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,606 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,616 - Recv: RTD MSB: 105 RTD LSB: 96 Fault: 0
2021-04-26 18:12:48,626 - Recv: RTD MSB: 105 RTD LSB: 96 Fault: 0
2021-04-26 18:12:48,636 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,647 - Recv: RTD MSB: 105 RTD LSB: 92 Fault: 0
2021-04-26 18:12:48,657 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,666 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,677 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,686 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,697 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,706 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,716 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,727 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,737 - Recv: RTD MSB: 105 RTD LSB: 78 Fault: 0
2021-04-26 18:12:48,747 - Recv: RTD MSB: 105 RTD LSB: 78 Fault: 0
2021-04-26 18:12:48,757 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,767 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,777 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,787 - Recv: RTD MSB: 105 RTD LSB: 86 Fault: 0
2021-04-26 18:12:48,797 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,807 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,808 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,817 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,827 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,837 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,847 - Recv: RTD MSB: 105 RTD LSB: 90 Fault: 0
2021-04-26 18:12:48,857 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,867 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,877 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,930 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0
2021-04-26 18:12:48,937 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,947 - Recv: RTD MSB: 105 RTD LSB: 88 Fault: 0
2021-04-26 18:12:48,970 - Recv: RTD MSB: 105 RTD LSB: 82 Fault: 0
2021-04-26 18:12:48,977 - Recv: RTD MSB: 105 RTD LSB: 82 Fault: 0
2021-04-26 18:12:48,987 - Recv: RTD MSB: 105 RTD LSB: 82 Fault: 0
2021-04-26 18:12:48,997 - Recv: RTD MSB: 105 RTD LSB: 84 Fault: 0

That's 92 times in a second. This basically happens because Marlin invokes readRTD() from within analog_to_celsius_hotend().

Also, THERMOCOUPLE_MAX_ERRORS is used only when calling read_max_tc(), but as you can see in temperature.cpp:

void Temperature::updateTemperaturesFromRawValues() {
  TERN_(TEMP_SENSOR_0_IS_MAX_TC, temp_hotend[0].raw = READ_MAX_TC(0));
  TERN_(TEMP_SENSOR_1_IS_MAX_TC, temp_hotend[1].raw = READ_MAX_TC(1));
  #if HAS_HOTEND
    HOTEND_LOOP() temp_hotend[e].celsius = analog_to_celsius_hotend(temp_hotend[e].raw, e);
  #endif

right after reading raw values, marlin re-reads the temperature from within analog_to_celsius_hotend(). And there is no error checking there, the temperature read is used as-is.

Obviously this is wrong and it should be averted.

Also, my solution has some assumptions (like a default value of 20°C is hardcoded as a default return value when there are no readings) which need to be fixed.

I do wonder though, why are you using 1-shot reading instead of auto-convert?

[zeleps]

Ok, I think whoever put this line there:

        #elif TEMP_SENSOR_0_IS_MAX_TC
          return TERN(TEMP_SENSOR_0_IS_MAX31865, max31865_0.temperature(MAX31865_SENSOR_OHMS_0, MAX31865_CALIBRATION_OHMS_0), raw * 0.25);

did not expect temperature() to re-read the RTD register. I've changed temperature() to use the lastRead value and everything looks much better now. I'll cleanup the code later and upload it to my fork.

[GadgetAngel]

Ok, I think whoever put this line there:

        #elif TEMP_SENSOR_0_IS_MAX_TC
          return TERN(TEMP_SENSOR_0_IS_MAX31865, max31865_0.temperature(MAX31865_SENSOR_OHMS_0, MAX31865_CALIBRATION_OHMS_0), raw * 0.25);

did not expect temperature() to re-read the RTD register. I've changed temperature() to use the lastRead value and everything looks much better now. I'll cleanup the code later and upload it to my fork.

MAX6675_HEAT_INTERVAL is the Marlin variable which decides how fast to do the ADC reading in Marlin. MAX6675_HEAT_INTERVAL is located on the second line in Temperature::read_max_tc() routine.

The reason I use one shot is because you do not need to constantly read the temperature every millisecond. Since the temperature readings are occurring slowly (as compared to real-time readings) I choose to use one shot. Also the original Adafruit library performed its temperature readings this way. They developed the MAX31865 board so I followed their lead.

Also look at the Marlin variable MAX_CONSECUTIVE_LOW_TEMPERATURE_ERROR_ALLOWED in temperature.cpp code around line 2609 - 2628.

I see what you are saying now, I believe you are correct, that re-reading the RTD register twice in a row should not occur.

But you should not be getting any faulty temperature readings from the MAX31865 sensor.

There has to be a combination of these changes that gives you the speed you need, and temperature readings that are not faulty.

Let me know when you solve it, I will use your library when you are done testing when I get my printer put back together. Also inform me if after you update your library if you get any faulty temperature readings. Thanks for these ideas.