Using Scrolling Text with Matrix Panel and #HHMM format in a Segment, Time gets wrong with -8 hour offset once in a while

[YvesLev]

What happened?

This happens once every day sometime once every few days.

This happens on 2 differents Led Panel setup (WS2812b) consisting of 16 X 32 leds.
It seems to happen less often on the Panel in the Kitchen, But this one has a Usermod (BH1750) and a Node-Red automation to auto dim (change the intensity) according to ambiant light. Also has MQTT enabled. But the problem exists also.

One segment has scrolling text effect to display the Time with #HHMM format in the Segment description.
Segment 2 display the outdoor temperature using REST command from HA.

The time gets wrong by itself. -8 Hour (-7h since daylight saving). It is not exactly 8/7 hour.

Exemple: Panel shows 10:40. Real time is 17:37.

It does not seems to correct by itself.
To correct the time, I go in the UI, click on info, I see the time is wrong, then, while in the info window, I changed the intensity, it corrects the time immediatly.
Going in any config menu and just pressing save with no modif. will correct the problem.
Also, re-selecting the preset that is already running will fix the time.

I'm using an ESP32-Mini on both clock panel.
My Time Zone US-EST/EDT.

To Reproduce Bug

I cannot reproduce the bug.
It just happen.
I notice it when I look at the clock.

Expected Behavior

WLED Internal Time gets off by about 8 hours (7 hours since daylight saving is active).

Install Method

Binary from WLED.me

What version of WLED?

0.14.0 Release

Which microcontroller/board are you seeing the problem on?

ESP32

Relevant log/trace output

No response

Anything else?

I'm using my local NTP server (Which is my Router).
Before I was using another Local NTP server (A windows workstation).
I just put back the default ntp server (0.wled.pool.ntp.org) and will report back.

Code of Conduct

• I agree to follow this project's Code of Conduct

[blazoncek]

Scrolling Text uses system time, so if a display is incorrect it means system time is incorrect.
System time is determined by querying NTP server. You should diagnose your NTP servers.

If there is no response for NTP server, WLED will keep its internal count of time which may get skewed with long running without NTP as the internal crystal frequency is not guaranteed to be precise.

[YvesLev]

Scrolling Text uses system time, so if a display is incorrect it means system time is incorrect. System time is determined by querying NTP server. You should diagnose your NTP servers.

I checked my NTP servers (Local ones) and they are working fines.
I've used the Windows NTP server for years for all my devices in the house and now the Router NTP server is also working very well and i still have the same problem in the latest version (I never saw this problem in 0.13.3 as far as I remember).
I changed yesterday the NTP server in wled for the default one (External) and we'll see.

If there is no response for NTP server, WLED will keep its internal count of time which may get skewed with long running without NTP as the internal crystal frequency is not guaranteed to be precise.

This is not the behaviour I observed since it suddenly jumps 8hrs in the past.
A glitch once in a while in NTP server should not make WLED to jump 8 hrs all of a sudden.
And eventually it should correct itself at the next ntp querying. (I just go in the UI and do some navigation (no changes) and then the time comes back to normal...)

What frequency or events make the queryings happening exactly?

There must be some checks in the code to prevent bad query responses (No response or too much gap from previous query should be discarded)?

Thanks

[softhack007]

A glitch once in a while in NTP server should not make WLED to jump 8 hrs all of a sudden.

Well it depends. Your NTP server should implement the NTP protocol, and only return a time response if it's own time queries do converge. WLED will use the server response as-is, and just subtract 1/2 of the "roundtrip time".
So if WLED time jumps, it could be a roundtrip time of 16hrs, which does not seem reasonable. Is your ntp server always online? Maybe you neded to use a different one.

What frequency or events make the queryings happening exactly?

Twice per day, with retries each 10 seconds (if no response).

There must be some checks in the code to prevent bad query responses (No response or too much gap from previous query should be discarded)?

Nope, a valid NTP response implies a valid time was provided from the server.

Relevant code is in wled00/ntp.cpp
https://github.com/Aircoookie/WLED/blob/1dab26bcbcac051f2b7be47a2d5c757a9938bf1f/wled00/ntp.cpp#L242

Please run a wled debug build, and use serial monitor (via USB) to track debug messages. If network time is the problem, this should be clearly visible in the debug output.

@blazoncek hi, I'm back :-)

[YvesLev]

Thank you for your help...

Yes my local NTP server is alway online as far as I know and have been used for many years with other devices (HA, Tasmota, etc)...

First it was a Windows PC running HA in a VM... Then a week ago I setup my wleds to use my Router local NTP server, But same problem happens.

Last night, I put back the original NTP server that comes with Wled fw. We'll see how it goes.

Also, I wonder why while the problem is there and when I go in the UI, info screen and change the main brightness slider, it corrects the time.
Does this action generate a NTP request in WLED?

Thanks for all the detailed answers.
I'll keep testing and report back.

Yves

[softhack007]

Does this action generate a NTP request in WLED?

Not directly, but possibly indirectly. UI changes trigger a call to stateUpdated(). This function triggers the webserver and other callbacks. As a consequence, the network kernel (async udp) might release a new receive operation, which then triggers our UDP receiver notify function. Due to this, the notifier might process a new NTP package and update local clocks.

But there is no direct generation of NTP requests on UI interactions.

[softhack007]

@YvesLev i've added some robustness improvements to our NTP client code.
Please try changes from PR #3536 , and report back.

[YvesLev]

@softhack007 thank you.

I have to figure out what I have to do to try it :-)
Do I have to compile the branch 0.14.1 to have those modifications ? or is it the branch ntp_errorchecking?
From what I see, ntp.cpp does not have recent changes in 0_14_1 branch.
So I guess this is the branch ntp_errorchecking.
(I'm not familiar to github yet...)

Ok just saw that 0_14_1 has been updated while I was updating my post :-)

Yves

[softhack007]

@YvesLev Yes I've just merged the changes into 0_14_1.

It's best if you build the firmware from source code, however it might also work to use one of the "artifacts" created by the automated github build: https://github.com/Aircoookie/WLED/actions/runs/6935308439, then scroll down to find the list of "artifacts".