Pause, and occasional failure with restart

[armyofevilrobots]

The pronterface log window fills with:

--- snip ---
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 ISR overtaking itself.
Error:0 I
--- snip ---

Then the printer fails with no further output, or occasionally restarts with:

Error:Line Number is not Last Line Number+1, Last Line:0
Resend:1

Also, it didn't crash, but earlier on I had the following:

--- snip ---
Error:32 ISR overtaking itself.
Error:32 ISR overtaking itself.
Error:32 ISR overtaking itself.
--- snip ---

In both cases, the logs have hundreds of lines like this, THEN usually continue with no problems.

Seems more common with tight corners and short segments. Related?

[bkubicek]

is this happening in-print? What is your platform / cpu speed? Could it be related to endstops being triggering when the ISR thing starts? How long does this take, maybe ~4 seconds? How often does this happen?
Is it reproducable, or random?

Thanks for the report!

[blende64]

I can confirm this. It happens while printing when a layer is almost finished. Printing stops some seconds while showing many "Error:0 ISR overtaking itself." Errors. Then it resumes with the next layer. After about 10-15 layers the firmware is crashed and doesn't respond anymore. (print was centered, far away from endstops)

Marlin 1.0.0 Beta 1 compiled with Arduino 0023
Orca v0.2 with Gen6
Pronterface Version unknown (mid/end October)
SFACT V42.4.1 11.09.15

[bkubicek]

once the ISR sends this routine, its likely that it will fail the next time, as serial sending blocks. Maybe it would recover faster if we send the message only once. It is not a solution.But it would be nice to know what causes this.

anyone, can you print the same file again, and is the error at the same location? Whats the frequency of the sanguinolulu?

ratenki: whats your microcontroller frequency, 16 mhz?

[armyofevilrobots]

I am on a ramps 1.3 with arduino mega128 @16mhz iirc. I won't try again on the same print (yoda, 12 hours into a 13 hour print), but I'll see if it happens with my fine detail test print and report back if it is consistently failing in the same spot.

I also have increased my hardware buffer from 16 to 32 blocks, which seemed to make it more robust, but didn't eliminate the problem (and I may be fooling myself, as I didn't do A/B testing).

[blende64]

I just tried again. It will fail reliable but at random points. One common behavior is that pronterface becomes very instable/crashed when printing stops. After restarting pronterface you can reconnect without restarting the printer. ( I restarted the printer befor every try to be sure)

1. try:
   starting from home position. It crashed after the first z-move ( lifting nozzle). Many "Error:28265 ISR overtaking itself." and then "endstop hit" but the opto is physically not triggered.
2. try:
   printed about 10 layers successfully. Then many "Error:28265 ISR overtaking itself." . Moved 1-2 cm and finally crashed with many "Error:28265 ISR overtaking itself."
3. try:
   after lifting the nozzle, many isr errors, then moving about 2cm and befor this move finished it crashed with this:
   " ...
   Error:26478 ISR overtaking itself.
   Error:26478 ISR overtaking itself.
   Error:26478 ISR overtaking itself.
   Error:
   echo:endstops hit: Z:1.91
   "
   no opto was physically triggered

electronic is standart mendel-parts Gen6

[armyofevilrobots]

Pronterface will block waiting for results, so that isn't terribly surprising. I have crashed it by holding the arduino down in reset or unplugging the usb. Since the firmware is crashing, this is kind of expected.

Interesting that the numbers we see ie Error:$integer ISR overtaking itself. are all different... /me off to read source.

[armyofevilrobots]

A little further examination of the code has me thinking we are lucky the heaters aren't running away.

Also, I added a local patch to only send the serial error once, instead of every overrun. I'll commit back if it helps.

Finally, I had the crash again, it happens at totally different places on every run for me.

[armyofevilrobots]

Re-ran with the following changes:

// in configuration.h
#define BUFSIZE 5
#define BLOCK_BUFFER_SIZE 32

// in stepper.cpp
volatile int errorsent=0;

//in ISR(TIMER1_COMPA_vect)
if(busy){
  if(errorsent){
    SERIAL_ERROR_START
    SERIAL_ERROR(*(unsigned short *)OCR1A);
    SERIAL_ERRORLNPGM(" ISR overtaking itself.");
    errorsent=1;
   }
   return; 
}
///etc the rest of ISR
errorsent=0;

This prevented duplicate ISR error sends. It doesn't do anything to address the root cause, although I wonder if a larger minimum segment size would prevent the overruns by merging multiple short moves? I currently have 5, but with 16x microstepping, that is < 1/10 of a mm on my machine. Perhaps I could do 10? Just spitballing...

[blende64]

@rantenki: Thank you, that's it.

side effect: On all tests before, the print was not centered as it should be. Now it's correctly centered!
...
my 2 hour print just finished successfully with no ISR Error.

[armyofevilrobots]

Unfortunately this fix isn't addressing the root cause, whatever it might be, although I suspect short segments resulting in the ISR taking longer than the inter-step time. We can probably figure that out with static analysis, but that doesn't sound like fun. ;) It isn't going to help people with 8Mhz and/or low ram (ie: atmega168) either.

[blende64]

But compared with a failed print this is a big step ahead. And with this fix it's for me more stable than the "official" 0.9.xx versions. ( improved resolution/quality would be really great too...)

[bkubicek]

I have been thinking a bit out what could be a possible reason for this. The only things i find that are not100% deterministic when printing idedntical files, concerning the ISR's operation, are the two following:

• the fill state of the buffer. If the the buffer runs empty,the velocities and accelerations can change. Also, in case of low buffer, the ISR might or might maybe take a different, longer branch. In this case, larger movement buffer could help a tiny bit, as ratenki might or might not have seen.
• Endstop hits or Endstop triggered for a very very short time due to electromagnetic noise. The latter would only be visible by lost microsteps= nearly not at all in normal print, if the firmware would only drop steps if the endstop is triggered. Counter this is the fact, that marlin then would stop the whole move, and this would lead to a probably visible shift. i honestly had one big layer shift, but i think that it was just a normal problem too-much-extrusion-bumping-into-the-rough-surface.
• It could maybe happen that the stepper ISR is delayed by the temperature-timer interrupt interrupt it, or maybe also a serial receive ISR, of which I honestly don't know the operational time.

If it is the timer, we could do some manual interrupt priorisation: an enum/define with values 0 1 2 that are stored in a volatile uint8_t. While operating the moveISR or tempISR writes a 1 or 2 respectively. After finishing, both write a 0.
Before starting the moveISR checks, if the uint8_t is active either by itself=very bad, or by the tempISR=continue. If the tempISR is called, it checks if the moveISR is active, and if so, returns. If this helps, also more clever things can be done, e.g. the tempISR counting how many times it did not go active, and telling the moveISR to let it finish for once after a nr of failures.

[ErikZalm]

blender64. You also seems to have a problem with your endstops. (False triggers)
I will change Marlin so that very small pulses will not be seens as endstop triggers. But you should also fix the problem in your printer.
Increase the distance between the endstop cables and the motor cables. Or even better use shielded cables.
ISR warning with a large number are probably caused by the endstops.

The ISR warning with 0 and 32 is a FW bug that I am investigating. The OCR1A should not be 0 or 32. I never write it with a value < 100. (A least not directly)

[ErikZalm]

rantentki,

Can you reproduce this error. One of the problems for me is that I can not reproduce it.

[bkubicek]

Erik: I have seen some problem that manifests as pause in print, but with autonomous sd card printing. I can try to have a Serial connected host software to monitor whats going on. I suspect it could be the same cause, so at least I can test. The annoying problem: it only happens for me 3 times in 3h cummulated print time.