Skip to content

lighttpd: lighttpd 1.4.79#90

Merged
Darchiv merged 1 commit intophoenix-rtos:masterfrom
gstrauss:lighttpd-1.4.77
Apr 18, 2025
Merged

lighttpd: lighttpd 1.4.79#90
Darchiv merged 1 commit intophoenix-rtos:masterfrom
gstrauss:lighttpd-1.4.77

Conversation

@gstrauss
Copy link
Contributor

@gstrauss gstrauss commented Jan 20, 2025

Description

lighttpd 1.4.79

Motivation and Context

modern lighttpd with bug fixes and HTTP/2 (optional)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Chore (refactoring, style fixes, git/CI config, submodule management, no code logic changes)
  • Breaking change: "mod_compress" needs to be renamed "mod_deflate" for static build.
  • Breaking change: the phoenix-rtos custom patch to look for precompressed files in mod_compress not longer applies, since mod_compress is no longer part of lighttpd. (lighttpd mod_compress has been subsumed by lighttpd mod_deflate.) Arbitrary content negotiation can be achieved using lighttpd mod_magnet and lua: https://wiki.lighttpd.net/AbsoLUAtion#Content-Negotiation

[Edit] These mod_compress -> mod_deflate breaking changes are less likely to affect many maintainers since https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf does not list mod_compress (or mod_deflate), and so it would have had to be added manually by the maintainer.

How Has This Been Tested?

  • Already covered by automatic testing.
  • New test added: (add PR link here).
  • Tested by hand on: (list targets here).

Tested by lighttpd CI on many platforms, but not on phoenix-rtos. Submitting this PR will test build on phoenix-rtos.

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing linter checks and tests passed.
  • My changes generate no new compilation warnings for any of the targets. (TBD)

Special treatment

  • This PR needs additional PRs to work (list the PRs, preferably in merge-order).
  • I will merge this PR by myself when appropriate.

@gstrauss
Copy link
Contributor Author

Future possible changes: lighttpd 1.4.56 and later add support for alternative TLS libraries. In addition to lighttpd mod_openssl, there is now lighttpd mod_gnutls, lighttpd mod_mbedtls, lighttpd mod_wolfssl, and lighttpd mod_nss.

For resource-constrained embedded systems, lighttpd mod_mbedtls using mbedtls, or lighttpd mod_wolfssl using wolfssl, is recommended due to having a smaller footprint than openssl. The lighttpd.conf syntax to configure lighttpd TLS modules is almost identical across TLS modules, with the most important exception being TLS-library-specific spellings of ciphers. If using lighttpd TLS defaults, this is a non-issue since a custom cipher list does not need to be specified in lighttpd.conf.

@gstrauss
Copy link
Contributor Author

lighttpd no longer ships configure, so autoreconf -fi must be run to generate configure from configure.ac.

I pushed a change to fix that in lighttpd/build.sh.

Note: It is interesting that many of the jobs in the CI matrix were successful (without attempting to lighttpd).
https://github.com/phoenix-rtos/phoenix-rtos-ports/actions/runs/12861205643

@agkaminski agkaminski requested a review from Darchiv January 21, 2025 15:38
@gstrauss
Copy link
Contributor Author

The CI fails here:

 /opt/phoenix/arm-phoenix/bin/../lib/gcc/arm-phoenix/9.5.0/../../../../arm-phoenix/bin/ld: lighttpd-plugin.o:(.rodata.load_functions+0x4c): undefined reference to `mod_evasive_plugin_init'
/opt/phoenix/arm-phoenix/bin/../lib/gcc/arm-phoenix/9.5.0/../../../../arm-phoenix/bin/ld: lighttpd-plugin.o:(.rodata.load_functions+0x6c): undefined reference to `mod_usertrack_plugin_init'

lighttpd mod_evasive and lighttpd mod_usertrack were removed in lighttpd 1.4.68, and can be replaced with lua scripts
https://wiki.lighttpd.net/ModMagnetExamples

Where is the CI getting lighttpd.conf from which it generates the static plugin list?
phoenix-rtos-ports lighttpd/build.sh contains: CONFIGFILE=$(find "${PREFIX_ROOTFS:?PREFIX_ROOTFS not set!}/etc" -name "lighttpd.conf"), but /etc/...lighttpd.conf is not present in phoenix-rtos-ports repository.

https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf
has mod_evasive and mod_usertrack commented out. That commit was May 12, 2020 and is the only commit for that file, so I don't think that is the file used by the CI here.

Related, why is that in phoenix-rtos/phoenix-rtos-project instead of part of phoenix-rtos/phoenix-rtos-ports? I would suggest improvements to that lighttpd.conf, but some improvements require a base version of lighttpd.


warning: comparison of integer expressions of different signedness: 'int' and 'time_t' {aka 'long long unsigned int'} [-Wsign-compare]

WTH? Why does phoenix-rtos set time_t to long long unsigned int? time_t is supposed to be a signed type. For the Y2038 problem, time_t has been changed from int to int64_t (or long long int) by any sane operating system.

@agkaminski
Copy link
Member

agkaminski commented Jan 21, 2025

warning: comparison of integer expressions of different signedness: 'int' and 'time_t' {aka 'long long unsigned int'} [-Wsign-compare]

WTH? Why does phoenix-rtos set time_t to long long unsigned int? time_t is supposed to be a signed type. For the Y2038 problem, time_t has been changed from int to int64_t (or long long int) by any sane operating system.

https://pubs.opengroup.org/onlinepubs/009695399/basedefs/time.h.html
https://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/types.h.html
"time_t and clock_t shall be integer or real-floating types."

That's actually a funny fact that the standard does not define signedness of time_t, so strictly speaking unsigned long long is a valid choice. It comes with many practical drawbacks though (lack of compatibility with everything else being one of them), so the change to the long long is already planned.

@gstrauss
Copy link
Contributor Author

Yes, you're correct that the sign of time_t is unspecified by the standards.

It comes with many practical drawbacks though

Yes.

Given that some time functions return -1 on error induces the need to cast (time_t)-1 to avoid pedantic warnings, and makes testing with comparison operators </> prone to error if -1 is cast to a very large unsigned value.

@gstrauss
Copy link
Contributor Author

@agkaminski Where is the CI getting lighttpd.conf from which it generates the static plugin list? That is what is causing the CI failure.

@agkaminski
Copy link
Member

agkaminski commented Jan 21, 2025

@agkaminski Where is the CI getting lighttpd.conf from which it generates the static plugin list? That is what is causing the CI failure.

Not really my piece of cake, but I see it only in the generic rootfs skel in the OS main repository:
https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf (edit - updated the link to master branch)

Regarding the warning - IMHO let it stay to further motivate time_t change ;)

@gstrauss
Copy link
Contributor Author

Regarding the warning - IMHO let it stay to further motivate time_t change ;)

Well, there is one place in lighttpd which fails due to underflow and wraparound of unsigned (where signed time_t was incorrectly expected), but it is in a less-used feature for mod_dirlisting caching.

/github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c: In function 'mod_dirlisting_cache_check':
Warning: /github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c:1719:17: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits]
 1719 |     if (max_age < 0)
      |                 ^

@agkaminski
Copy link
Member

Regarding the warning - IMHO let it stay to further motivate time_t change ;)

Well, there is one place in lighttpd which fails due to underflow and wraparound of unsigned (where signed time_t was incorrectly expected), but it is in a less-used feature for mod_dirlisting caching.

/github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c: In function 'mod_dirlisting_cache_check':
Warning: /github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c:1719:17: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits]
 1719 |     if (max_age < 0)
      |                 ^

Thank you for pointing out this issue and for the PR in general

@gstrauss
Copy link
Contributor Author

As you and I pointed out, https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf is the likely origin of mod_evasive and mod_usertrack in the CI. lighttpd/build.sh greps out mod_ from the file, which includes commented-out lines. I'll submit a PR to that repo, and once accepted there, will hopefully unblock this PR.

If @Darchiv would like to switch from lighttpd mod_openssl to lighttpd mod_mbedtls for lower resource use, then the CI will run into this issue again since the lighttpd.conf is in a separate repo and is not matched in lock-step with the lighttpd build configuration in this repo.

@gstrauss
Copy link
Contributor Author

@Darchiv lighttpd/build.sh might contain a list of lighttpd modules to build, rather than grep from lighttpd.conf in a different repository. Besides the example above for moving from mod_openssl to mod_mbedtls, if you want to provide HTTP/2 support, then mod_h2 needs to be listed.

lighttpd-git pushed a commit to lighttpd/lighttpd1.4 that referenced this pull request Jan 23, 2025
signedness of time_t is not specified by the standards,
though most sane systems use a signed type for time_t.

phoenix-rtos currently defines time_t as unsigned long long int

x-ref:
  phoenix-rtos/phoenix-rtos-ports#90
  phoenix-rtos/phoenix-rtos-project#1261
@Darchiv
Copy link
Member

Darchiv commented Jan 24, 2025

As you and I pointed out, https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf is the likely origin of mod_evasive and mod_usertrack in the CI. lighttpd/build.sh greps out mod_ from the file, which includes commented-out lines. I'll submit a PR to that repo, and once accepted there, will hopefully unblock this PR.

If @Darchiv would like to switch from lighttpd mod_openssl to lighttpd mod_mbedtls for lower resource use, then the CI will run into this issue again since the lighttpd.conf is in a separate repo and is not matched in lock-step with the lighttpd build configuration in this repo.

The phoenix-rtos-project repository contains a reference project, which is the fastest way to start using Phoenix-RTOS. A custom project can use this repository (via forking or any other way) and override configuration stored in root-skel. Hence, if a project needs mod_mbedtls, it can use a custom lighttpd.conf and the upstream phoenix-rtos-project is not relevant anymore.

@Darchiv lighttpd/build.sh might contain a list of lighttpd modules to build, rather than grep from lighttpd.conf in a different repository. Besides the example above for moving from mod_openssl to mod_mbedtls, if you want to provide HTTP/2 support, then mod_h2 needs to be listed.

Maybe module listing can be improved, but it still has to be configurable per-project - for example by an env variable exported by the project. This can be more cumbersome and error-prone than just using an up-to-date lighttpd.conf, though.


Have you checked that mod_wstunnel still builds? You have removed its patch, but I assume that autogen.sh will properly include it into the build.

[Edit] These mod_compress -> mod_deflate breaking changes are less likely to affect many maintainers since https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf does not list mod_compress (or mod_deflate), and so it would have had to be added manually by the maintainer.

Employing lua just for this small mod_compress fix-up may be too much of a burden on an embedded system. Maybe there should be an (optional) way to patch mod_deflate in the same manner as mod_compress.

Just pointing out the not-so-obvious use cases we have to cover. Your PR is nevertheless valuable.

@gstrauss
Copy link
Contributor Author

@Darchiv lighttpd/build.sh might contain a list of lighttpd modules to build, rather than grep from lighttpd.conf in a different repository. Besides the example above for moving from mod_openssl to mod_mbedtls, if you want to provide HTTP/2 support, then mod_h2 needs to be listed.

Maybe module listing can be improved, but it still has to be configurable per-project - for example by an env variable exported by the project. This can be more cumbersome and error-prone than just using an up-to-date lighttpd.conf, though.

There are many approaches to do this. lighttpd provides the mechanism plugin_static.h and uses C macros. phoenix-rtos chose to use a fragile grep pipeline with a naive 'parse' of lighttpd.conf, and results in building modules that are commented out in the reference lighttpd.conf. One alternative: phoenix-rtos could have provided a reference plugin_static.h which was simply copied into the project to be built. I'm sure there are many other valid ways. Since the way phoenix-rtos does it is with grep, please consider adding a line to better document "this is phoenix-rtos-provided convenience mechanism to generate plugin_static.h" in lighttpd/build.sh.

Please review phoenix-rtos/phoenix-rtos-project#1262, which is a prerequisite to this PR.

Have you checked that mod_wstunnel still builds? You have removed its patch, but I assume that autogen.sh will properly include it into the build.

lighttpd CI builds lighttpd on many platforms (phoenix-rtos is not part of lighttpd CI) and I have made many improvements to lighttpd since the 1.4.53 version hard-coded in the phoenix-rtos reference build. If mod_wstunnel does not build on phoenix-rtos, then I would like to fix the issue upstream, if possible. The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

[Edit] These mod_compress -> mod_deflate breaking changes are less likely to affect many maintainers since https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf does not list mod_compress (or mod_deflate), and so it would have had to be added manually by the maintainer.

Employing lua just for this small mod_compress fix-up may be too much of a burden on an embedded system. Maybe there should be an (optional) way to patch mod_deflate in the same manner as mod_compress.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos? As you said, this is a reference build. Please share the reasons why this was added and why, for a reference build, this is a required patch before you accept this PR ("Just pointing out the not-so-obvious use cases we have to cover.") The customization does not fix any bugs. It adds a feature, and does so in a very specific way by breaking/replacing the existing mod_compress caching logic. Are you planning to port that to lighttpd mod_deflate caching?

If someone developing embedded systems is using lighttpd, then as a lighttpd developer, I recommend the documentation:
https://wiki.lighttpd.net/Docs_ResourceTuning
https://wiki.lighttpd.net/Docs_Performance

If there is a need for Content Negotiation for pre-compressed files and if an embedded system has low resources, then I would recommend omitting mod_deflate and zlib and instead using lighttpd mod_magnet and lua. You'll likely find the disk, memory footprint, and CPU usage to be similar for Content-Negotiation for pre-compressed files.

If Content Negotiation for pre-compressed files must be in C on resource-constrained embedded systems, then I still recommend omitting mod_deflate, and instead writing a custom module based on lighttpd the simple mod_staticfile.c to perform the specific Content Negotation for pre-compressed files.

@gstrauss
Copy link
Contributor Author

ping.

The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos?

1 similar comment
@gstrauss
Copy link
Contributor Author

gstrauss commented Mar 6, 2025

ping.

The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos?

@gstrauss
Copy link
Contributor Author

ping.

The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos?

@gstrauss
Copy link
Contributor Author

ping.

The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos?

With the merge of phoenix-rtos/phoenix-rtos-project#1262 (thank you) and a minor update to build.sh, the CI on this PR now passes.

Similar to phoenix-rtos/phoenix-rtos-project#1262, this PR cleans up plenty of technical debt, removing all patches that were previously added to lighttpd by phoenix-rtos-project.

@Darchiv
Copy link
Member

Darchiv commented Apr 1, 2025

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos? As you said, this is a reference build. Please share the reasons why this was added and why, for a reference build, this is a required patch before you accept this PR ("Just pointing out the not-so-obvious use cases we have to cover.") The customization does not fix any bugs. It adds a feature, and does so in a very specific way by breaking/replacing the existing mod_compress caching logic. Are you planning to port that to lighttpd mod_deflate caching?

These tickets are for internal projects and are not to be viewed by people not part of the phoenix-rtos team. You are right that customizations should not alter the core software or official modules, especially since this is a port and not a fork. We will probably create a separate custom module (entirely in C) for pre-compressed files, as you have suggested.

When it comes to plugin_static.h and grepping modules, changing the approach is out-of-scope of this PR. I think we can get back to this issue once we update lighttpd.

Before merging, please resolve the following issues:

  1. You have recently updated lighttpd to 1.4.78 in build.sh, but please also update PR's title, contents, and commit message accordingly.
  2. Version 1.4.78 includes a fix of missing headers (lighttpd/lighttpd1.4@385c7dd). We weren't sure that there will be a lighttpd release soon, so we bypassed the error by introducing -Wno-error=implicit-function-declaration: build: adapt to gcc 14.2 #85 (comment) Could you please rebase your changes and remove this flag, so that no implicit-function-declaration warnings will be missed in the future?

* update to lighttpd 1.4.79
* do not error out if patches/*.patch do not exist
* revert part of 6cb7bac build: adapt to gcc 14.2 (JIRA: RTOS-927)
  which modified lighttpd/build.sh to add
  -Wno-error=implicit-function-declaration
* run ./autogen.sh to autoreconf -fi before ./configure lighttpd build
@gstrauss gstrauss changed the title lighttpd: lighttpd 1.4.77 lighttpd: lighttpd 1.4.79 Apr 4, 2025
@gstrauss
Copy link
Contributor Author

gstrauss commented Apr 4, 2025

Apologies for my delayed response. I wanted to get a bug-fix included in the lighttpd 1.4.79 release.

Before merging, please resolve the following issues:

You have recently updated lighttpd to 1.4.78 in build.sh, but please also update PR's title, contents, and commit message accordingly.

Done.

Version 1.4.78 includes a fix of missing headers (lighttpd/lighttpd1.4@385c7dd). We weren't sure that there will be a lighttpd release soon, so we bypassed the error by introducing -Wno-error=implicit-function-declaration:

Done. I removed that change. As you noted there, it would have been better as a standalone patch taken from upstream rather than modifying the compiler flags. (Also, I would have patched it sooner had the compilation portability issue been reported upstream.)

These tickets are for internal projects and are not to be viewed by people not part of the phoenix-rtos team. You are right that customizations should not alter the core software or official modules, especially since this is a port and not a fork. We will probably create a separate custom module (entirely in C) for pre-compressed files, as you have suggested.

As I can not read minds, and can not read tickets to which I have no access, I do not have the context or requirements behind that feature. Still, I highly recommend taking a look at https://wiki.lighttpd.net/mod_magnet and
https://wiki.lighttpd.net/AbsoLUAtion#Content-Negotiation contains some customizable lua.
The lua solution customized to a specific type (e.g. .zstd) will be faster than a generic solution which tries a list (e.g. .gz, .zstd) where the first try or tries do not exist. Excess stat() syscalls are much slower than content negotiation code written in either Lua or C. Yes, there is an initial memory cost to adding lighttpd mod_magnet and supporting lua libs to lighttpd. On a 32-bit ARM device I have, adding mod_magnet added < 150KB to the RSS of my lighttpd process. That can very flexibly support custom lua scripts and without any maintenance burden of a custom C module.

@gstrauss
Copy link
Contributor Author

gstrauss commented Apr 4, 2025

(#85 Reference to the patch introducing -Wno-error=implicit-function-declaration solved upstream with lighttpd/lighttpd1.4@385c7dd)


For the future, if you are interested in reducing memory and disk usage, please see https://wiki.lighttpd.net/Docs_SSL and evaluate lighttpd mod_mbedtls or lighttpd mod_wolfssl which use TLS libraries aimed at embedded systems, rather than lighttpd mod_openssl which relies on the larger openssl libraries.

Copy link
Member

@Darchiv Darchiv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my nitpicks. Everything compiles nicely, so I'm merging the upgrade now.

for patchfile in "${PREFIX_PORT}"/patches/*.patch; do
if [ ! -f "$patchfile" ]; then
continue;
fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: in phoenix-rtos-ports/build.subr we have common helper functions, currently only download and patch functions, which are meant to be more sophisticated and cover most build use cases in ports. See phoenix-rtos-ports/lua/build.sh for an example usage. Not all ports were rewritten to use these functions (yet).

Rewriting this build.sh is probably out of scope of this PR, but feel free to do it later.

@Darchiv Darchiv merged commit 6cc3c34 into phoenix-rtos:master Apr 18, 2025
36 checks passed
@gstrauss gstrauss mentioned this pull request Apr 19, 2025
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants