[Request] Force browsers to reload web app

[schlimmchen]

Is your feature request related to a problem? Please describe.

Web applications incompatible with the firmware keep causing issues.

Describe the solution you'd like

I know there are ways to make the browser fetch a new copy of the web app. I used this in some application some time ago. The idea is to request the resources with an URL parameter, and the backend will only answer with the respective HTTP response code if the parameter matches the backend firmware version. Something like that... Not sure if that still works, though. Modern browsers might be even more "smart".

Describe alternatives you've considered

No response

Additional context

No response

[schlimmchen]

@tbnobody Is this something you explored already? If so, you might have insight regarding the feasibility. If not, I think I want to have a look at this, as telling users to clear their browser cache and such, or even having these problems in the first place, becomes cumbersome.

[tbnobody]

I don't know why it happens.... the etag header is sent, a full reload is performed after firmware upgrade. I don't know why some browsers seem to ignore it..

The idea is to request the resources with an URL parameter, and the backend will only answer with the respective HTTP response code if the parameter matches the backend firmware version

The ETag header does exactly this.

[schlimmchen]

Had to look at the code, as I was unaware that you put something like that in already.

I agree, that should do it. At least because you also put in a Header "Cache-Control", "public, must-revalidate".

Using what I have in mind is a nightmare... The idea is that rather than js/app.js, js/app.js?v=<git sha> is requested.

Oooooh, wait! index.html often (mostly?) is the same among releases, right? Since all the other resources are merely loaded through index.html, the browser might actually be checking for a new version of index.html, as it should, but it's the same, so the important resources like js/app.js are not actually reloaded. The browser assumes that they are also still valid, as index.html did not change...

Also, assuming that the etag is not only working, but actually used (using it is optional, as far as I can tell), I think there is an easy solution. Let me draft a PR and I will mention you.

[tbnobody]

Just doublechecked it... Even if I just press F5 (so without clearing any cache) the browser asks for the app.js and gets the right answer (in this case 304)

the page reload is executed here (imho there is no way to say the browser to ignore the cache)
https://github.com/tbnobody/OpenDTU/blob/f0db2a51daf349fa9ae7919883b6a9ea279b85dc/webapp/src/views/WaitRestartView.vue#L50

[schlimmchen]

Let's assume #1898 fixed this. We will have to observe this for a couple of months after the next release.

[stefan123t]

For ETag's to work, doesn't the Web Server (in this case the ESP32AsyncWebServer) need to support the HEAD request, in order for the client browser to be able to request the current (updated with the git-hash) ETag's whilst revalidating the resources.

The workaround that @schlimmchen suggested of adding a changed version to the Request URL parameters did work fine for several PWA Applications on older Browser versions. This should still work if the HEAD request together with the git-hash unique ETag's does not solve the issue.

[schlimmchen]

Looks good (enough) to me:

I did not know about the HEAD request before, and now I am confused that this answer does contain content? Well, the OpenDTU webserver does support HEAD, even it is sends back too much (the content).

[stefan123t]

Yes, IMO the HEAD request was introduced together with the WebDAV extensions RFC-2518 until its latest incarnation as RFC-4918, but has also seen widespread use-cases with other technologies, like Caching / Proxies, etc.

In principle it allows a client to check whether a Resource / URL and the "data collection" / Folder behind it has changed its (WebDAV) content. Before requesting the complete content it requests only the HTTP HEADers, which can then be checked by comparing the ETag header and the If-None-Match Headers on either/both Server and/or Client side before responding with the full Body or requesting only a partial update, e.g. a single changed block from byte X to Y, as e.g. used in Resuming Downloads and/or some Block Filesystems available via HTTP transport (think e.g. Dropbox).

Details of `ETag` and `If-None-Match`

The ESP8266WebServer describes it as follows and last made changes in PR #8227:

When enabled by default the server reads the file content and creates a checksum using the md5 and base64 algorithm her called the ETag value that changes whenever the file contains something different.
Once a browser has got the content of a file from the server including the ETag information it will add that ETag value is added in the following requests for the same resource. Now the server can answer with a 'use the version from the cache' when the new calculated ETag value is equal to the ETag value in the request.
The calculation of the ETag value requires some time and processing but sending content is always slower. So when you have the situation that a browser will use a web server multiple times this mechanism saves network and computing and makes web pages more responsive.

Namely the following change was made amongst others to the libraries/ESP8266WebServer/src/detail/RequestHandlersImpl.h

            if (server.header("If-None-Match") == eTagCode) {
                server.send(304);
                return true;
            }
...
        if (SRH::_cache_header.length() != 0)
            server.sendHeader("Cache-Control", SRH::_cache_header);


        if ((server._eTagEnabled) && (_eTagCode.length() > 0)) {
            server.sendHeader("ETag", _eTagCode);
        }

Here is another eplanation BCsabaEngine/svelteesp32 that uses the PsychicHttpServer or ESP32AsyncWebServer to serve Svelte, ReactJS, Vue or Angular apps from an ESP32:

ETag
The ETag HTTP header can be used to significantly reduce network traffic. If the server sends ETag information, the client can check the integrity of the file by sending back this ETag (in If-None-Match header) without sending the data back again. All browsers use this function, the 304 HTTP response code is clearly visible in the network traffic.
Since microcontroller data traffic is moderately expensive, it is an individual decision whether to use the ETag or not. We recommend using ETag, which adds a bit more code (about 1-3%) but results in a much cleaner operation.
The use of ETag is not enabled by default, ...

Cache-control
By default (when using the ETag), we send no-cache in the Cache-Control header of the HTTP response. Pages, subpages and other elements are downloaded every time. This is perfectly acceptable when serving small pages with ESP.
At the same time, it can be an advantage that the content is cached by the browser and not even the ETag check is performed. For this, you can specify how many seconds the max-age value sent instead of no-cache should be. In the case of --cachetime=86400 (max-age=86400), the page (and other elements) will not be downloaded by the browser for one day.

I found the pictures on Werner Rothschopfs page about Arduino WebServer Caching with ESP8266 very instructive.

So coming back to OpenDTU and the ESP32AsyncWebServer which answers to your HEAD request with a full GET response body.
This would not be necessary and could be spared from the precious CPU cycles of our ESP32-S3 or whatever the code is running on.

Here is a request together with an If-None-Match header with the current ETag which is returning the expected 304 Not Modified Response

$ curl -v -H 'If-None-Match: "22a34b79871413b73b71840875cadb58"' http://opendtu-fusion/js/app.js --output app.js.gz 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Host opendtu-fusion:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.xxx.xxx
*   Trying 192.168.xxx.xxx:80...
* Connected to opendtu-fusion (192.168.xxx.xxx) port 80
> GET /js/app.js HTTP/1.1
> Host: opendtu-fusion
> User-Agent: curl/8.5.0
> Accept: */*
> If-None-Match: "22a34b79871413b73b71840875cadb58"
> 
< HTTP/1.1 304 Not Modified
< connection: close
< Cache-Control: public, must-revalidate
< ETag: "22a34b79871413b73b71840875cadb58"
< accept-ranges: none
< content-length: 0
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection

I have tried to check how it is actually implemented in the incarnation of the ESP32AsyncWebServer which we are using in OpenDTU.

More Details about the `HTTP_HEAD` Handler in mathieucarbou/ESP32AsyncWebServer

Here is the If-None-Match Handler code in WebHandlers.cpp

  // if-none-match has precedence over if-modified-since
  if (request->hasHeader(T_INM)) {
    not_modified = request->header(T_INM).equals(etag);
  } else if (_last_modified.length()) {
    not_modified = request->header(T_IMS).equals(_last_modified);
  }
...
  response->addHeader(T_ETag, etag.c_str());

  if (_last_modified.length()) {
    response->addHeader(T_Last_Modified, _last_modified.c_str());
  }
  if (_cache_control.length()) {
    response->addHeader(T_Cache_Control, _cache_control.c_str());
  }

and here is a stub for resuming Downloads using the HTTP_HEAD WebRequestMethod in ResumableDownload.ino:

  /*
    ❯ curl -I -X HEAD http://192.168.4.1/download
    HTTP/1.1 200 OK
    Content-Length: 1024
    Content-Type: application/octet-stream
    Connection: close
    Accept-Ranges: bytes
  */
  // Ref: https://github.com/mathieucarbou/ESPAsyncWebServer/pull/80
  server.on("/download", HTTP_HEAD | HTTP_GET, [](AsyncWebServerRequest *request) {
    if (request->method() == HTTP_HEAD) {
      AsyncWebServerResponse *response = request->beginResponse(200, "application/octet-stream");
      response->addHeader(asyncsrv::T_Accept_Ranges, "bytes");
      response->addHeader(asyncsrv::T_Content_Length, 10);
      response->setContentLength(1024);  // make sure we can overrides previously set content length
      response->addHeader(asyncsrv::T_Content_Type, "foo");
      response->setContentType("application/octet-stream");  // make sure we can overrides previously set content type
      // ...
      request->send(response);
    } else {
      // ...
    }
  });

and the ChunkedResponse.ino

  // first time: serves the file and cache headers
  // curl -N -v http://192.168.4.1/ --output -
  //
  // secodn time: serves 304
  // curl -N -v -H "if-none-match: 4272" http://192.168.4.1/ --output -
  //
  server.on("/", HTTP_GET, [](AsyncWebServerRequest *request) {
    String etag = String(htmlContentLength);


    if (request->header(asyncsrv::T_INM) == etag) {
      request->send(304);
      return;
    }

So from my understanding we may still have some work to do in order to resolve the Browser issue.
But first we should get a HAR (HTTP Archive) to analyse the situation when someone experiences this issue again.
So anyone who faces this issue should reload the Live View page using F12 > Network, reloading the OpenDTU Homepage e.g. using F5 or Reload button and then use the Download HAR archive button to attach it to this or either a new github issue.

[stefan123t]

Also note that we are returning a different ETag hash-sum "1e878dc7185fcc444783a57f38861dc8" when responding to the HEAD request rather than in the case of a GET request "22a34b79871413b73b71840875cadb58" in excess of the actually superfluous gzipped content being returned in the case of a HEAD request too.

$ curl -v -I -H 'If-None-Match: "22a34b79871413b73b71840875cadb58"' http://opendtu-fusion/js/app.js
* Host opendtu-fusion:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.xxx.xxx
*   Trying 192.168.xxx.xxx:80...
* Connected to opendtu-fusion (192.168.xxx.xxx) port 80
> HEAD /js/app.js HTTP/1.1
> Host: opendtu-fusion
> User-Agent: curl/8.5.0
> Accept: */*
> If-None-Match: "22a34b79871413b73b71840875cadb58"
> 
< HTTP/1.1 200 OK
< Content-Encoding: gzip
< Cache-Control: public, must-revalidate
< ETag: "1e878dc7185fcc444783a57f38861dc8"
< connection: close
< accept-ranges: none
< content-length: 392
< content-type: text/html

< 
* Excess found: excess = 392 url = /js/app.js (zero-length body)
* Closing connection

Actually it returns the gzipped index.html.gz file instead as a response to the /js/app.js URL with a HEAD request, which has indeed the ETag: "1e878dc7185fcc444783a57f38861dc8"

$ curl -v -H 'If-None-Match: "22a34b79871413b73b71840875cadb58"' http://opendtu-fusion/ --output index.html.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host opendtu-fusion:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.xxx.xxx
*   Trying 192.168.xxx.xxx:80...
* Connected to opendtu-fusion (192.168.xxx.xxx) port 80
> GET / HTTP/1.1
> Host: opendtu-fusion
> User-Agent: curl/8.5.0
> Accept: */*
> If-None-Match: "22a34b79871413b73b71840875cadb58"
> 
< HTTP/1.1 200 OK
< Content-Encoding: gzip
< Cache-Control: public, must-revalidate
< ETag: "1e878dc7185fcc444783a57f38861dc8"
< connection: close
< accept-ranges: none
< content-length: 392
< content-type: text/html
< 
{ [392 bytes data]
100   392  100   392    0     0  24846      0 --:--:-- --:--:-- --:--:-- 26133
* Closing connection

Together with the following three caveats from the above depicted explanation by Werner Rothschopf this may be the cause for the oddities we see on some devices / browsers:

• The value of the ETag has to be sent in double quotes.
• Even if you send the ETag with double quotes you can't rely that a browser sends back the value in double quotes. I see several browsers which are NOT returning the ETag in double quotes (for example the OEM Samsung browser) which makes the server comparison more difficult.
• Don't rely on that you will receive the HTTP Header field exactly as "If-None-Match", be prepared to get it in small or capital letters also.

When returning the ETag value here "22a34b79871413b73b71840875cadb58" with double quotes in the If-None-Match header without the double quotes, will deliver the gzipped content again.

Details for missing double quotes in the If-None-Match request header

$ curl -v -H 'If-None-Match: 22a34b79871413b73b71840875cadb58' http://opendtu-fusion/js/app.js --output app.js.gz 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Host opendtu-fusion:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.xxx.xxx
*   Trying 192.168.xxx.xxx:80...
* Connected to opendtu-fusion (192.168.xxx.xxx) port 80
> GET /js/app.js HTTP/1.1
> Host: opendtu-fusion
> User-Agent: curl/8.5.0
> Accept: */*
> If-None-Match: 22a34b79871413b73b71840875cadb58
> 
< HTTP/1.1 200 OK
< Content-Encoding: gzip
< Cache-Control: public, must-revalidate
< ETag: "22a34b79871413b73b71840875cadb58"
< connection: close
< accept-ranges: none
< content-length: 233529
< content-type: text/javascript
< 
{ [1216 bytes data]
100  228k  100  228k    0     0   632k      0 --:--:-- --:--:-- --:--:--  631k
* Closing connection

[stefan123t]

@schlimmchen maybe you could verify my observations with Wireshark again, but it looks like we are returning the index.html.gz with the HEAD request to the /js/app.js URL instead of just the ETag for the gzipped /js/app.js.gz without a body.

[schlimmchen]

Thanks, @stefan123t for your work here. As far as I can see now, all your points valid. Would you please be so kind and create a new issue where you collect all of the issues you found, so they are tracked? I will assign myself and handle this.